[==========] Running 26 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 8 tests from TxnWriteOpsITest
[ RUN ] TxnWriteOpsITest.CommitTimestampPropagation
2025-10-28T14:51:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:51:28Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251028 14:51:28.482704 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37607
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37607
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20251028 14:51:28.901109 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:28.901504 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:28.901578 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:28.911335 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:51:28.911463 424 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:51:28.911510 424 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20251028 14:51:28.911545 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:28.911607 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:51:28.911652 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:51:28.923257 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37607
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:28.925057 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:28.927361 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:28.940198 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:28.940274 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:28.940224 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:28.940625 424 server_base.cc:1047] running on GCE node
I20251028 14:51:28.942057 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:28.944380 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:28.948047 424 hybrid_clock.cc:648] HybridClock initialized: now 1761663088947832 us; error 167 us; skew 500 ppm
I20251028 14:51:28.948562 424 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:28.951651 424 webserver.cc:492] Webserver started at http://127.0.104.62:39437/ using document root <none> and password file <none>
I20251028 14:51:28.952589 424 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:28.952733 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:28.953127 424 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:28.957754 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c"
format_stamp: "Formatted at 2025-10-28 14:51:28 on dist-test-slave-0v9m"
I20251028 14:51:28.959113 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c"
format_stamp: "Formatted at 2025-10-28 14:51:28 on dist-test-slave-0v9m"
I20251028 14:51:28.966998 424 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20251028 14:51:28.972546 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:28.974848 424 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20251028 14:51:28.975113 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c"
format_stamp: "Formatted at 2025-10-28 14:51:28 on dist-test-slave-0v9m"
I20251028 14:51:28.975370 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:28.996685 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:28.997774 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:28.998080 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:29.027542 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37607
I20251028 14:51:29.027932 499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37607 every 8 connection(s)
I20251028 14:51:29.029381 424 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:51:29.034970 500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:29.036825 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 424
I20251028 14:51:29.037187 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:51:29.048985 500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c: Bootstrap starting.
I20251028 14:51:29.053081 500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:29.054859 500 log.cc:826] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:29.059602 500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c: No bootstrap required, opened a new log
I20251028 14:51:29.067989 500 raft_consensus.cc:359] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37607 } }
I20251028 14:51:29.068567 500 raft_consensus.cc:385] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:29.068703 500 raft_consensus.cc:740] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573fa50dd1ad4e3a8535c7c8bcd3a64c, State: Initialized, Role: FOLLOWER
I20251028 14:51:29.069587 500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37607 } }
I20251028 14:51:29.069953 500 raft_consensus.cc:399] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:51:29.070122 500 raft_consensus.cc:493] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:51:29.070291 500 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:29.074276 500 raft_consensus.cc:515] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37607 } }
I20251028 14:51:29.075006 500 leader_election.cc:304] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 573fa50dd1ad4e3a8535c7c8bcd3a64c; no voters:
I20251028 14:51:29.075793 500 leader_election.cc:290] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:51:29.076084 505 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:29.077023 505 raft_consensus.cc:697] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [term 1 LEADER]: Becoming Leader. State: Replica: 573fa50dd1ad4e3a8535c7c8bcd3a64c, State: Running, Role: LEADER
I20251028 14:51:29.077718 505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37607 } }
I20251028 14:51:29.078868 500 sys_catalog.cc:565] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:51:29.081806 506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37607 } } }
I20251028 14:51:29.082669 506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [sys.catalog]: This master's current role is: LEADER
I20251028 14:51:29.084590 507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 573fa50dd1ad4e3a8535c7c8bcd3a64c. Latest consensus state: current_term: 1 leader_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "573fa50dd1ad4e3a8535c7c8bcd3a64c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37607 } } }
I20251028 14:51:29.085186 507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c [sys.catalog]: This master's current role is: LEADER
I20251028 14:51:29.087107 514 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:51:29.094556 514 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:51:29.108793 514 catalog_manager.cc:1357] Generated new cluster ID: 2daa33ccef1e4ad5bd8b8a5052591705
I20251028 14:51:29.108989 514 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:51:29.128149 514 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:51:29.130194 514 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:51:29.144408 514 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 573fa50dd1ad4e3a8535c7c8bcd3a64c: Generated new TSK 0
I20251028 14:51:29.145711 514 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:51:29.162565 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37607
--builtin_ntp_servers=127.0.104.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:51:29.160276 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35340:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:51:29.171161 500 master.cc:464] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20251028 14:51:29.641938 529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:29.642316 529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:29.642426 529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:29.652268 529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:29.652459 529 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:29.652580 529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:51:29.663498 529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:37607
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:29.665364 529 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:29.667650 529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:29.679965 535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:29.679940 534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:29.680732 537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:29.681743 529 server_base.cc:1047] running on GCE node
I20251028 14:51:29.682680 529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:29.683815 529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:29.685119 529 hybrid_clock.cc:648] HybridClock initialized: now 1761663089685032 us; error 60 us; skew 500 ppm
I20251028 14:51:29.685525 529 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:29.688939 529 webserver.cc:492] Webserver started at http://127.0.104.1:38505/ using document root <none> and password file <none>
I20251028 14:51:29.689877 529 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:29.690009 529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:29.690385 529 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:29.693207 529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "670992dbf0c74b05a5c779f9a1e05dcc"
format_stamp: "Formatted at 2025-10-28 14:51:29 on dist-test-slave-0v9m"
I20251028 14:51:29.694159 529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "670992dbf0c74b05a5c779f9a1e05dcc"
format_stamp: "Formatted at 2025-10-28 14:51:29 on dist-test-slave-0v9m"
I20251028 14:51:29.700665 529 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20251028 14:51:29.704774 543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:29.706673 529 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:51:29.706915 529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "670992dbf0c74b05a5c779f9a1e05dcc"
format_stamp: "Formatted at 2025-10-28 14:51:29 on dist-test-slave-0v9m"
I20251028 14:51:29.707139 529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:29.734966 529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:29.735949 529 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:29.736397 529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:29.740898 529 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:29.741063 529 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:29.741201 529 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:29.741314 529 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20251028 14:51:29.812767 529 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45893
I20251028 14:51:29.812826 663 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45893 every 8 connection(s)
I20251028 14:51:29.814929 529 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:51:29.818231 664 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37607
I20251028 14:51:29.818609 664 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:29.819567 664 heartbeater.cc:507] Master 127.0.104.62:37607 requested a full tablet report, sending...
I20251028 14:51:29.820541 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 529
I20251028 14:51:29.820847 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:51:29.822412 454 ts_manager.cc:194] Registered new tserver with Master: 670992dbf0c74b05a5c779f9a1e05dcc (127.0.104.1:45893)
I20251028 14:51:29.824362 454 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:43811
I20251028 14:51:29.826622 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37607
--builtin_ntp_servers=127.0.104.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:51:30.216961 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35344:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:51:30.291563 666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:30.291952 666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:30.292093 666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:30.301477 666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:30.301817 666 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:30.301976 666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:51:30.314811 666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:37607
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:30.317401 666 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:30.320351 666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:30.335772 676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:30.335789 677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:30.336207 679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:30.338480 666 server_base.cc:1047] running on GCE node
I20251028 14:51:30.339603 666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:30.341194 666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:30.342554 666 hybrid_clock.cc:648] HybridClock initialized: now 1761663090342474 us; error 50 us; skew 500 ppm
I20251028 14:51:30.343020 666 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:30.346175 666 webserver.cc:492] Webserver started at http://127.0.104.2:34461/ using document root <none> and password file <none>
I20251028 14:51:30.347107 666 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:30.347255 666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:30.347637 666 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:30.350632 666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "6bf726a14de8444ea39dc15fa6c9950f"
format_stamp: "Formatted at 2025-10-28 14:51:30 on dist-test-slave-0v9m"
I20251028 14:51:30.351594 666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "6bf726a14de8444ea39dc15fa6c9950f"
format_stamp: "Formatted at 2025-10-28 14:51:30 on dist-test-slave-0v9m"
I20251028 14:51:30.359378 666 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20251028 14:51:30.365428 685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:30.367612 666 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20251028 14:51:30.367871 666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "6bf726a14de8444ea39dc15fa6c9950f"
format_stamp: "Formatted at 2025-10-28 14:51:30 on dist-test-slave-0v9m"
I20251028 14:51:30.368112 666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:30.402774 666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:30.403790 666 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:30.404111 666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:30.409358 666 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:30.409572 666 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:30.409723 666 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:30.409830 666 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:30.483697 666 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:36779
I20251028 14:51:30.483870 804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:36779 every 8 connection(s)
I20251028 14:51:30.485518 666 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:51:30.488883 805 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37607
I20251028 14:51:30.489374 805 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:30.490687 805 heartbeater.cc:507] Master 127.0.104.62:37607 requested a full tablet report, sending...
I20251028 14:51:30.492954 454 ts_manager.cc:194] Registered new tserver with Master: 6bf726a14de8444ea39dc15fa6c9950f (127.0.104.2:36779)
I20251028 14:51:30.493894 454 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:46715
I20251028 14:51:30.494385 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 666
I20251028 14:51:30.494570 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:51:30.499400 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37607
--builtin_ntp_servers=127.0.104.20:37271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:51:30.828904 664 heartbeater.cc:499] Master 127.0.104.62:37607 was elected leader, sending a full tablet report...
W20251028 14:51:30.904827 807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:30.905191 807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:30.905328 807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:30.916088 807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:30.916250 807 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:30.916365 807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:51:30.926868 807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:37607
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:30.928998 807 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:30.931102 807 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:30.945087 813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:30.945290 812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:30.947165 815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:30.947432 807 server_base.cc:1047] running on GCE node
I20251028 14:51:30.948139 807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:30.949447 807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:30.951045 807 hybrid_clock.cc:648] HybridClock initialized: now 1761663090950939 us; error 45 us; skew 500 ppm
I20251028 14:51:30.951709 807 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:30.956684 807 webserver.cc:492] Webserver started at http://127.0.104.3:37035/ using document root <none> and password file <none>
I20251028 14:51:30.958232 807 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:30.958505 807 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:30.959268 807 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:30.964715 807 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "02a18a3caef1435980f0168312ee2aa3"
format_stamp: "Formatted at 2025-10-28 14:51:30 on dist-test-slave-0v9m"
I20251028 14:51:30.965801 807 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "02a18a3caef1435980f0168312ee2aa3"
format_stamp: "Formatted at 2025-10-28 14:51:30 on dist-test-slave-0v9m"
I20251028 14:51:30.973546 807 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20251028 14:51:30.980564 821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:30.983361 807 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20251028 14:51:30.983604 807 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "02a18a3caef1435980f0168312ee2aa3"
format_stamp: "Formatted at 2025-10-28 14:51:30 on dist-test-slave-0v9m"
I20251028 14:51:30.983879 807 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:30.999953 807 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:31.000895 807 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:31.001188 807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:31.004971 807 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:31.005146 807 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:31.005286 807 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:31.005385 807 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251028 14:51:31.074611 941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:41295 every 8 connection(s)
I20251028 14:51:31.074503 807 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:41295
I20251028 14:51:31.076324 807 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:51:31.079578 942 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37607
I20251028 14:51:31.080248 942 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:31.081696 942 heartbeater.cc:507] Master 127.0.104.62:37607 requested a full tablet report, sending...
I20251028 14:51:31.083971 454 ts_manager.cc:194] Registered new tserver with Master: 02a18a3caef1435980f0168312ee2aa3 (127.0.104.3:41295)
I20251028 14:51:31.084964 454 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:55341
I20251028 14:51:31.085232 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 807
I20251028 14:51:31.085595 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:51:31.092073 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:51:31.120159 454 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35354:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:51:31.121315 454 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:51:31.153789 739 tablet_service.cc:1505] Processing CreateTablet for tablet 59d8dda9e69146d9aaa48aa5886c8469 (DEFAULT_TABLE table=txn_write_ops_test [id=bbd3a7f92b584aed9ee48c2bcd0e29b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:31.155143 740 tablet_service.cc:1505] Processing CreateTablet for tablet cc286291deb34e44b86fd9f5445b5ec2 (DEFAULT_TABLE table=txn_write_ops_test [id=bbd3a7f92b584aed9ee48c2bcd0e29b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:31.156180 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc286291deb34e44b86fd9f5445b5ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.156073 598 tablet_service.cc:1505] Processing CreateTablet for tablet 59d8dda9e69146d9aaa48aa5886c8469 (DEFAULT_TABLE table=txn_write_ops_test [id=bbd3a7f92b584aed9ee48c2bcd0e29b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:31.156407 877 tablet_service.cc:1505] Processing CreateTablet for tablet cc286291deb34e44b86fd9f5445b5ec2 (DEFAULT_TABLE table=txn_write_ops_test [id=bbd3a7f92b584aed9ee48c2bcd0e29b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:31.157694 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc286291deb34e44b86fd9f5445b5ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.157685 598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d8dda9e69146d9aaa48aa5886c8469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.161537 599 tablet_service.cc:1505] Processing CreateTablet for tablet cc286291deb34e44b86fd9f5445b5ec2 (DEFAULT_TABLE table=txn_write_ops_test [id=bbd3a7f92b584aed9ee48c2bcd0e29b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:31.161491 876 tablet_service.cc:1505] Processing CreateTablet for tablet 59d8dda9e69146d9aaa48aa5886c8469 (DEFAULT_TABLE table=txn_write_ops_test [id=bbd3a7f92b584aed9ee48c2bcd0e29b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:31.162206 599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc286291deb34e44b86fd9f5445b5ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.163436 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d8dda9e69146d9aaa48aa5886c8469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.168100 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d8dda9e69146d9aaa48aa5886c8469. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.180353 959 tablet_bootstrap.cc:492] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3: Bootstrap starting.
I20251028 14:51:31.186556 960 tablet_bootstrap.cc:492] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc: Bootstrap starting.
I20251028 14:51:31.187216 959 tablet_bootstrap.cc:654] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.187433 961 tablet_bootstrap.cc:492] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f: Bootstrap starting.
I20251028 14:51:31.189121 959 log.cc:826] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:31.191095 960 tablet_bootstrap.cc:654] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.191628 961 tablet_bootstrap.cc:654] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.192657 960 log.cc:826] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:31.192800 961 log.cc:826] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:31.198904 961 tablet_bootstrap.cc:492] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f: No bootstrap required, opened a new log
I20251028 14:51:31.198904 960 tablet_bootstrap.cc:492] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc: No bootstrap required, opened a new log
I20251028 14:51:31.199246 960 ts_tablet_manager.cc:1403] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20251028 14:51:31.199246 961 ts_tablet_manager.cc:1403] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20251028 14:51:31.199312 959 tablet_bootstrap.cc:492] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3: No bootstrap required, opened a new log
I20251028 14:51:31.199602 959 ts_tablet_manager.cc:1403] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.000s
I20251028 14:51:31.205291 959 raft_consensus.cc:359] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.205721 959 raft_consensus.cc:385] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.205835 959 raft_consensus.cc:740] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a18a3caef1435980f0168312ee2aa3, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.206530 959 consensus_queue.cc:260] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.206871 960 raft_consensus.cc:359] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.206871 961 raft_consensus.cc:359] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.207295 960 raft_consensus.cc:385] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.207357 961 raft_consensus.cc:385] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.207415 960 raft_consensus.cc:740] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670992dbf0c74b05a5c779f9a1e05dcc, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.207443 961 raft_consensus.cc:740] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf726a14de8444ea39dc15fa6c9950f, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.208143 960 consensus_queue.cc:260] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.208156 961 consensus_queue.cc:260] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.212993 805 heartbeater.cc:499] Master 127.0.104.62:37607 was elected leader, sending a full tablet report...
I20251028 14:51:31.213194 959 ts_tablet_manager.cc:1434] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20251028 14:51:31.213407 942 heartbeater.cc:499] Master 127.0.104.62:37607 was elected leader, sending a full tablet report...
I20251028 14:51:31.213560 961 ts_tablet_manager.cc:1434] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20251028 14:51:31.213821 959 tablet_bootstrap.cc:492] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3: Bootstrap starting.
I20251028 14:51:31.214128 961 tablet_bootstrap.cc:492] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f: Bootstrap starting.
I20251028 14:51:31.215807 959 tablet_bootstrap.cc:654] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.215834 961 tablet_bootstrap.cc:654] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.220543 960 ts_tablet_manager.cc:1434] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc: Time spent starting tablet: real 0.021s user 0.010s sys 0.003s
I20251028 14:51:31.221243 960 tablet_bootstrap.cc:492] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc: Bootstrap starting.
I20251028 14:51:31.223349 960 tablet_bootstrap.cc:654] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.234210 961 tablet_bootstrap.cc:492] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f: No bootstrap required, opened a new log
I20251028 14:51:31.234292 960 tablet_bootstrap.cc:492] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc: No bootstrap required, opened a new log
I20251028 14:51:31.234432 959 tablet_bootstrap.cc:492] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3: No bootstrap required, opened a new log
I20251028 14:51:31.234458 960 ts_tablet_manager.cc:1403] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20251028 14:51:31.234593 959 ts_tablet_manager.cc:1403] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.009s
I20251028 14:51:31.234609 961 ts_tablet_manager.cc:1403] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.003s
I20251028 14:51:31.235363 960 raft_consensus.cc:359] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.235390 959 raft_consensus.cc:359] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.235682 959 raft_consensus.cc:385] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.235680 960 raft_consensus.cc:385] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.235811 959 raft_consensus.cc:740] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a18a3caef1435980f0168312ee2aa3, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.235848 960 raft_consensus.cc:740] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670992dbf0c74b05a5c779f9a1e05dcc, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.235934 961 raft_consensus.cc:359] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.236246 961 raft_consensus.cc:385] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.236371 961 raft_consensus.cc:740] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf726a14de8444ea39dc15fa6c9950f, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.236388 959 consensus_queue.cc:260] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.236388 960 consensus_queue.cc:260] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.236891 961 consensus_queue.cc:260] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.237901 961 ts_tablet_manager.cc:1434] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251028 14:51:31.238687 806 tablet.cc:2378] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:51:31.239069 959 ts_tablet_manager.cc:1434] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20251028 14:51:31.240917 960 ts_tablet_manager.cc:1434] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20251028 14:51:31.254230 455 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35368:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:51:31.254974 455 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:51:31.263474 740 tablet_service.cc:1505] Processing CreateTablet for tablet cd3fa13b295942d192b019bd421c2280 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d60f42a6caf44eccb5a74ac58ba1e46f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:51:31.263173 599 tablet_service.cc:1505] Processing CreateTablet for tablet cd3fa13b295942d192b019bd421c2280 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d60f42a6caf44eccb5a74ac58ba1e46f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:51:31.263784 876 tablet_service.cc:1505] Processing CreateTablet for tablet cd3fa13b295942d192b019bd421c2280 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d60f42a6caf44eccb5a74ac58ba1e46f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:51:31.264251 599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd3fa13b295942d192b019bd421c2280. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.264245 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd3fa13b295942d192b019bd421c2280. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.264545 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd3fa13b295942d192b019bd421c2280. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:31.271998 959 tablet_bootstrap.cc:492] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: Bootstrap starting.
I20251028 14:51:31.272683 960 tablet_bootstrap.cc:492] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: Bootstrap starting.
I20251028 14:51:31.272899 961 tablet_bootstrap.cc:492] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: Bootstrap starting.
I20251028 14:51:31.274212 959 tablet_bootstrap.cc:654] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.274600 960 tablet_bootstrap.cc:654] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.274801 961 tablet_bootstrap.cc:654] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:31.277490 959 tablet_bootstrap.cc:492] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: No bootstrap required, opened a new log
I20251028 14:51:31.277729 959 ts_tablet_manager.cc:1403] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251028 14:51:31.278899 959 raft_consensus.cc:359] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.279211 959 raft_consensus.cc:385] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.279301 959 raft_consensus.cc:740] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02a18a3caef1435980f0168312ee2aa3, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.279661 959 consensus_queue.cc:260] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.280221 965 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.280504 965 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:31.281694 959 ts_tablet_manager.cc:1434] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:51:31.281754 960 tablet_bootstrap.cc:492] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: No bootstrap required, opened a new log
I20251028 14:51:31.281940 960 ts_tablet_manager.cc:1403] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20251028 14:51:31.282639 961 tablet_bootstrap.cc:492] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: No bootstrap required, opened a new log
I20251028 14:51:31.283044 961 ts_tablet_manager.cc:1403] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251028 14:51:31.283128 960 raft_consensus.cc:359] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.283370 960 raft_consensus.cc:385] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.283452 960 raft_consensus.cc:740] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 670992dbf0c74b05a5c779f9a1e05dcc, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.283748 960 consensus_queue.cc:260] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.284291 960 ts_tablet_manager.cc:1434] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 14:51:31.284293 967 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.284633 967 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:31.285328 961 raft_consensus.cc:359] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.285692 961 raft_consensus.cc:385] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:31.285796 961 raft_consensus.cc:740] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf726a14de8444ea39dc15fa6c9950f, State: Initialized, Role: FOLLOWER
I20251028 14:51:31.286142 961 consensus_queue.cc:260] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.286598 966 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.286916 966 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:31.288061 961 ts_tablet_manager.cc:1434] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251028 14:51:31.304991 967 raft_consensus.cc:493] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:31.305336 967 raft_consensus.cc:515] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.306066 966 raft_consensus.cc:493] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:31.306391 966 raft_consensus.cc:515] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.308049 967 leader_election.cc:290] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bf726a14de8444ea39dc15fa6c9950f (127.0.104.2:36779), 02a18a3caef1435980f0168312ee2aa3 (127.0.104.3:41295)
I20251028 14:51:31.308336 966 leader_election.cc:290] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 670992dbf0c74b05a5c779f9a1e05dcc (127.0.104.1:45893), 02a18a3caef1435980f0168312ee2aa3 (127.0.104.3:41295)
W20251028 14:51:31.318890 665 tablet.cc:2378] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:51:31.328269 943 tablet.cc:2378] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:51:31.333906 760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d8dda9e69146d9aaa48aa5886c8469" candidate_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf726a14de8444ea39dc15fa6c9950f" is_pre_election: true
I20251028 14:51:31.334447 760 raft_consensus.cc:2468] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 670992dbf0c74b05a5c779f9a1e05dcc in term 0.
I20251028 14:51:31.335667 546 leader_election.cc:304] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 670992dbf0c74b05a5c779f9a1e05dcc, 6bf726a14de8444ea39dc15fa6c9950f; no voters:
I20251028 14:51:31.336385 967 raft_consensus.cc:2804] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:31.336592 967 raft_consensus.cc:493] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:31.336750 967 raft_consensus.cc:3060] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.336722 897 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d8dda9e69146d9aaa48aa5886c8469" candidate_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a18a3caef1435980f0168312ee2aa3" is_pre_election: true
I20251028 14:51:31.337400 897 raft_consensus.cc:2468] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 670992dbf0c74b05a5c779f9a1e05dcc in term 0.
I20251028 14:51:31.338915 897 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc286291deb34e44b86fd9f5445b5ec2" candidate_uuid: "6bf726a14de8444ea39dc15fa6c9950f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a18a3caef1435980f0168312ee2aa3" is_pre_election: true
I20251028 14:51:31.339134 897 raft_consensus.cc:2468] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf726a14de8444ea39dc15fa6c9950f in term 0.
I20251028 14:51:31.339603 688 leader_election.cc:304] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a18a3caef1435980f0168312ee2aa3, 6bf726a14de8444ea39dc15fa6c9950f; no voters:
I20251028 14:51:31.339803 619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc286291deb34e44b86fd9f5445b5ec2" candidate_uuid: "6bf726a14de8444ea39dc15fa6c9950f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" is_pre_election: true
I20251028 14:51:31.340086 966 raft_consensus.cc:2804] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:31.340267 619 raft_consensus.cc:2468] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf726a14de8444ea39dc15fa6c9950f in term 0.
I20251028 14:51:31.340446 967 raft_consensus.cc:515] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.340890 966 raft_consensus.cc:493] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:31.341035 966 raft_consensus.cc:3060] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.341235 967 leader_election.cc:290] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [CANDIDATE]: Term 1 election: Requested vote from peers 6bf726a14de8444ea39dc15fa6c9950f (127.0.104.2:36779), 02a18a3caef1435980f0168312ee2aa3 (127.0.104.3:41295)
I20251028 14:51:31.341794 760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d8dda9e69146d9aaa48aa5886c8469" candidate_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf726a14de8444ea39dc15fa6c9950f"
I20251028 14:51:31.341862 897 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d8dda9e69146d9aaa48aa5886c8469" candidate_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a18a3caef1435980f0168312ee2aa3"
I20251028 14:51:31.342002 760 raft_consensus.cc:3060] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.342057 897 raft_consensus.cc:3060] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.344607 760 raft_consensus.cc:2468] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 670992dbf0c74b05a5c779f9a1e05dcc in term 1.
I20251028 14:51:31.344743 966 raft_consensus.cc:515] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.345377 546 leader_election.cc:304] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 670992dbf0c74b05a5c779f9a1e05dcc, 6bf726a14de8444ea39dc15fa6c9950f; no voters:
I20251028 14:51:31.345817 967 raft_consensus.cc:2804] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:31.345896 897 raft_consensus.cc:2468] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 670992dbf0c74b05a5c779f9a1e05dcc in term 1.
I20251028 14:51:31.346553 967 raft_consensus.cc:697] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [term 1 LEADER]: Becoming Leader. State: Replica: 670992dbf0c74b05a5c779f9a1e05dcc, State: Running, Role: LEADER
I20251028 14:51:31.347436 966 leader_election.cc:290] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [CANDIDATE]: Term 1 election: Requested vote from peers 670992dbf0c74b05a5c779f9a1e05dcc (127.0.104.1:45893), 02a18a3caef1435980f0168312ee2aa3 (127.0.104.3:41295)
I20251028 14:51:31.347611 619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc286291deb34e44b86fd9f5445b5ec2" candidate_uuid: "6bf726a14de8444ea39dc15fa6c9950f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670992dbf0c74b05a5c779f9a1e05dcc"
I20251028 14:51:31.347645 897 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc286291deb34e44b86fd9f5445b5ec2" candidate_uuid: "6bf726a14de8444ea39dc15fa6c9950f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02a18a3caef1435980f0168312ee2aa3"
I20251028 14:51:31.347501 967 consensus_queue.cc:237] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.347836 619 raft_consensus.cc:3060] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.347864 897 raft_consensus.cc:3060] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.351074 897 raft_consensus.cc:2468] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf726a14de8444ea39dc15fa6c9950f in term 1.
I20251028 14:51:31.351752 619 raft_consensus.cc:2468] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf726a14de8444ea39dc15fa6c9950f in term 1.
I20251028 14:51:31.351770 688 leader_election.cc:304] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a18a3caef1435980f0168312ee2aa3, 6bf726a14de8444ea39dc15fa6c9950f; no voters:
I20251028 14:51:31.352326 966 raft_consensus.cc:2804] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:31.353413 966 raft_consensus.cc:697] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [term 1 LEADER]: Becoming Leader. State: Replica: 6bf726a14de8444ea39dc15fa6c9950f, State: Running, Role: LEADER
I20251028 14:51:31.354009 966 consensus_queue.cc:237] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.354418 454 catalog_manager.cc:5649] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc reported cstate change: term changed from 0 to 1, leader changed from <none> to 670992dbf0c74b05a5c779f9a1e05dcc (127.0.104.1). New cstate: current_term: 1 leader_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } health_report { overall_health: UNKNOWN } } }
I20251028 14:51:31.359117 965 raft_consensus.cc:493] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:31.359335 965 raft_consensus.cc:515] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.360929 454 catalog_manager.cc:5649] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bf726a14de8444ea39dc15fa6c9950f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "6bf726a14de8444ea39dc15fa6c9950f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } health_report { overall_health: UNKNOWN } } }
I20251028 14:51:31.361202 965 leader_election.cc:290] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 670992dbf0c74b05a5c779f9a1e05dcc (127.0.104.1:45893), 6bf726a14de8444ea39dc15fa6c9950f (127.0.104.2:36779)
I20251028 14:51:31.369367 760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd3fa13b295942d192b019bd421c2280" candidate_uuid: "02a18a3caef1435980f0168312ee2aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf726a14de8444ea39dc15fa6c9950f" is_pre_election: true
I20251028 14:51:31.369639 760 raft_consensus.cc:2468] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a18a3caef1435980f0168312ee2aa3 in term 0.
I20251028 14:51:31.370157 824 leader_election.cc:304] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a18a3caef1435980f0168312ee2aa3, 6bf726a14de8444ea39dc15fa6c9950f; no voters:
I20251028 14:51:31.370280 619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd3fa13b295942d192b019bd421c2280" candidate_uuid: "02a18a3caef1435980f0168312ee2aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" is_pre_election: true
I20251028 14:51:31.370522 619 raft_consensus.cc:2468] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02a18a3caef1435980f0168312ee2aa3 in term 0.
I20251028 14:51:31.370543 965 raft_consensus.cc:2804] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:31.370644 965 raft_consensus.cc:493] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:31.370776 965 raft_consensus.cc:3060] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.372946 965 raft_consensus.cc:515] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.373601 965 leader_election.cc:290] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [CANDIDATE]: Term 1 election: Requested vote from peers 670992dbf0c74b05a5c779f9a1e05dcc (127.0.104.1:45893), 6bf726a14de8444ea39dc15fa6c9950f (127.0.104.2:36779)
I20251028 14:51:31.374202 619 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd3fa13b295942d192b019bd421c2280" candidate_uuid: "02a18a3caef1435980f0168312ee2aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "670992dbf0c74b05a5c779f9a1e05dcc"
I20251028 14:51:31.374197 760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd3fa13b295942d192b019bd421c2280" candidate_uuid: "02a18a3caef1435980f0168312ee2aa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bf726a14de8444ea39dc15fa6c9950f"
I20251028 14:51:31.374409 760 raft_consensus.cc:3060] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.374408 619 raft_consensus.cc:3060] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:31.377071 619 raft_consensus.cc:2468] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a18a3caef1435980f0168312ee2aa3 in term 1.
I20251028 14:51:31.377285 760 raft_consensus.cc:2468] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02a18a3caef1435980f0168312ee2aa3 in term 1.
I20251028 14:51:31.377733 823 leader_election.cc:304] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02a18a3caef1435980f0168312ee2aa3, 670992dbf0c74b05a5c779f9a1e05dcc; no voters:
I20251028 14:51:31.378104 965 raft_consensus.cc:2804] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:31.378549 965 raft_consensus.cc:697] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [term 1 LEADER]: Becoming Leader. State: Replica: 02a18a3caef1435980f0168312ee2aa3, State: Running, Role: LEADER
I20251028 14:51:31.379125 965 consensus_queue.cc:237] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } }
I20251028 14:51:31.383239 989 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: TxnStatusTablet state changed. Reason: New leader 02a18a3caef1435980f0168312ee2aa3. Latest consensus state: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.383581 989 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:31.385989 455 catalog_manager.cc:5649] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02a18a3caef1435980f0168312ee2aa3 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } health_report { overall_health: HEALTHY } } }
I20251028 14:51:31.393821 991 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:51:31.394212 991 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:51:31.454589 619 raft_consensus.cc:1275] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc [term 1 FOLLOWER]: Refusing update from remote peer 02a18a3caef1435980f0168312ee2aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:31.454702 760 raft_consensus.cc:1275] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f [term 1 FOLLOWER]: Refusing update from remote peer 02a18a3caef1435980f0168312ee2aa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:31.455943 965 consensus_queue.cc:1048] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:31.456568 989 consensus_queue.cc:1048] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:31.467432 966 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: TxnStatusTablet state changed. Reason: New leader 02a18a3caef1435980f0168312ee2aa3. Latest consensus state: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.467717 966 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:31.470074 995 mvcc.cc:204] Tried to move back new op lower bound from 7215772022587183104 to 7215772022293049344. Current Snapshot: MvccSnapshot[applied={T|T < 7215772022587183104}]
I20251028 14:51:31.471715 966 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.471995 966 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 6bf726a14de8444ea39dc15fa6c9950f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:31.475190 965 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.475498 965 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:31.479748 967 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: TxnStatusTablet state changed. Reason: New leader 02a18a3caef1435980f0168312ee2aa3. Latest consensus state: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.480029 967 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:31.491844 989 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.492152 989 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 02a18a3caef1435980f0168312ee2aa3: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:31.498430 996 mvcc.cc:204] Tried to move back new op lower bound from 7215772022587183104 to 7215772022293049344. Current Snapshot: MvccSnapshot[applied={T|T < 7215772022587183104}]
I20251028 14:51:31.498646 967 tablet_replica.cc:442] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02a18a3caef1435980f0168312ee2aa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 } } peers { permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 } } peers { permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 } } }
I20251028 14:51:31.499050 967 tablet_replica.cc:445] T cd3fa13b295942d192b019bd421c2280 P 670992dbf0c74b05a5c779f9a1e05dcc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:31.598621 897 raft_consensus.cc:1275] T 59d8dda9e69146d9aaa48aa5886c8469 P 02a18a3caef1435980f0168312ee2aa3 [term 1 FOLLOWER]: Refusing update from remote peer 670992dbf0c74b05a5c779f9a1e05dcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:31.600090 984 consensus_queue.cc:1048] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:31.602505 760 raft_consensus.cc:1275] T 59d8dda9e69146d9aaa48aa5886c8469 P 6bf726a14de8444ea39dc15fa6c9950f [term 1 FOLLOWER]: Refusing update from remote peer 670992dbf0c74b05a5c779f9a1e05dcc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:31.603442 984 consensus_queue.cc:1048] T 59d8dda9e69146d9aaa48aa5886c8469 P 670992dbf0c74b05a5c779f9a1e05dcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf726a14de8444ea39dc15fa6c9950f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:31.610085 619 raft_consensus.cc:1275] T cc286291deb34e44b86fd9f5445b5ec2 P 670992dbf0c74b05a5c779f9a1e05dcc [term 1 FOLLOWER]: Refusing update from remote peer 6bf726a14de8444ea39dc15fa6c9950f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:31.610631 896 raft_consensus.cc:1275] T cc286291deb34e44b86fd9f5445b5ec2 P 02a18a3caef1435980f0168312ee2aa3 [term 1 FOLLOWER]: Refusing update from remote peer 6bf726a14de8444ea39dc15fa6c9950f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:31.611584 986 consensus_queue.cc:1048] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [LEADER]: Connected to new peer: Peer: permanent_uuid: "670992dbf0c74b05a5c779f9a1e05dcc" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:31.612944 986 consensus_queue.cc:1048] T cc286291deb34e44b86fd9f5445b5ec2 P 6bf726a14de8444ea39dc15fa6c9950f [LEADER]: Connected to new peer: Peer: permanent_uuid: "02a18a3caef1435980f0168312ee2aa3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:31.643905 1012 mvcc.cc:204] Tried to move back new op lower bound from 7215772023174696960 to 7215772022165917696. Current Snapshot: MvccSnapshot[applied={T|T < 7215772023174696960}]
I20251028 14:51:32.386530 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 529
I20251028 14:51:32.387817 659 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:32.511292 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 529
I20251028 14:51:32.531672 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 666
I20251028 14:51:32.533494 800 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:32.671618 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 666
I20251028 14:51:32.697722 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 807
I20251028 14:51:32.699072 937 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:32.827904 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 807
W20251028 14:51:32.848145 441 connection.cc:537] server connection from 127.0.104.3:55341 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:51:32.848413 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 424
I20251028 14:51:32.849907 495 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:32.968376 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 424
2025-10-28T14:51:32Z chronyd exiting
[ OK ] TxnWriteOpsITest.CommitTimestampPropagation (4575 ms)
[ RUN ] TxnWriteOpsITest.DeadlockPrevention
2025-10-28T14:51:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:51:33Z Disabled control of system clock
I20251028 14:51:33.035017 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35435
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35435
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20251028 14:51:33.380914 1044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:33.381275 1044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:33.381350 1044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:33.389659 1044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:51:33.389782 1044 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:51:33.389839 1044 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20251028 14:51:33.389875 1044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:33.389935 1044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:51:33.389981 1044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:51:33.400125 1044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35435
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:33.401862 1044 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:33.403754 1044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:33.415163 1050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:33.415264 1049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:33.416806 1052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:33.417219 1044 server_base.cc:1047] running on GCE node
I20251028 14:51:33.418210 1044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:33.420120 1044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:33.421617 1044 hybrid_clock.cc:648] HybridClock initialized: now 1761663093421539 us; error 82 us; skew 500 ppm
I20251028 14:51:33.422147 1044 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:33.425433 1044 webserver.cc:492] Webserver started at http://127.0.104.62:38233/ using document root <none> and password file <none>
I20251028 14:51:33.426715 1044 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:33.426875 1044 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:33.427575 1044 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:33.430596 1044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "0543f36933ba4812802e875de2334f95"
format_stamp: "Formatted at 2025-10-28 14:51:33 on dist-test-slave-0v9m"
I20251028 14:51:33.431742 1044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "0543f36933ba4812802e875de2334f95"
format_stamp: "Formatted at 2025-10-28 14:51:33 on dist-test-slave-0v9m"
I20251028 14:51:33.439534 1044 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20251028 14:51:33.443783 1058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:33.445561 1044 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20251028 14:51:33.445771 1044 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "0543f36933ba4812802e875de2334f95"
format_stamp: "Formatted at 2025-10-28 14:51:33 on dist-test-slave-0v9m"
I20251028 14:51:33.446062 1044 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:33.489754 1044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:33.490763 1044 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:33.491050 1044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:33.515172 1044 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35435
I20251028 14:51:33.515220 1119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35435 every 8 connection(s)
I20251028 14:51:33.516789 1044 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:51:33.521420 1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:33.526613 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 1044
I20251028 14:51:33.526841 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:51:33.533103 1120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95: Bootstrap starting.
I20251028 14:51:33.537230 1120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:33.538796 1120 log.cc:826] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:33.542423 1120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95: No bootstrap required, opened a new log
I20251028 14:51:33.549572 1120 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0543f36933ba4812802e875de2334f95" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35435 } }
I20251028 14:51:33.550069 1120 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:33.550182 1120 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0543f36933ba4812802e875de2334f95, State: Initialized, Role: FOLLOWER
I20251028 14:51:33.550833 1120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0543f36933ba4812802e875de2334f95" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35435 } }
I20251028 14:51:33.551064 1120 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:51:33.551191 1120 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:51:33.551350 1120 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:33.553922 1120 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0543f36933ba4812802e875de2334f95" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35435 } }
I20251028 14:51:33.554600 1120 leader_election.cc:304] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0543f36933ba4812802e875de2334f95; no voters:
I20251028 14:51:33.555287 1120 leader_election.cc:290] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:51:33.555447 1125 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:33.556176 1125 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [term 1 LEADER]: Becoming Leader. State: Replica: 0543f36933ba4812802e875de2334f95, State: Running, Role: LEADER
I20251028 14:51:33.556806 1125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0543f36933ba4812802e875de2334f95" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35435 } }
I20251028 14:51:33.557654 1120 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:51:33.560576 1127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0543f36933ba4812802e875de2334f95. Latest consensus state: current_term: 1 leader_uuid: "0543f36933ba4812802e875de2334f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0543f36933ba4812802e875de2334f95" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35435 } } }
I20251028 14:51:33.560974 1127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [sys.catalog]: This master's current role is: LEADER
I20251028 14:51:33.561411 1126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0543f36933ba4812802e875de2334f95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0543f36933ba4812802e875de2334f95" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35435 } } }
I20251028 14:51:33.561694 1126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95 [sys.catalog]: This master's current role is: LEADER
I20251028 14:51:33.564374 1134 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:51:33.570399 1134 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:51:33.583793 1134 catalog_manager.cc:1357] Generated new cluster ID: 4537904476fd4fcda5af13b0d6c94424
I20251028 14:51:33.583983 1134 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:51:33.597915 1134 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:51:33.599382 1134 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:51:33.613196 1134 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0543f36933ba4812802e875de2334f95: Generated new TSK 0
I20251028 14:51:33.614188 1134 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:51:33.624961 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35435
--builtin_ntp_servers=127.0.104.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:51:33.623632 1075 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34974:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:51:33.633677 1120 master.cc:464] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20251028 14:51:34.033241 1149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:34.033645 1149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:34.033771 1149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:34.044418 1149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:34.044629 1149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:34.044948 1149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:51:34.056258 1149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:34.058101 1149 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:34.060173 1149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:34.071161 1155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:34.071041 1154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:34.072392 1157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:34.072775 1149 server_base.cc:1047] running on GCE node
I20251028 14:51:34.073571 1149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:34.074637 1149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:34.076092 1149 hybrid_clock.cc:648] HybridClock initialized: now 1761663094075853 us; error 64 us; skew 500 ppm
I20251028 14:51:34.076473 1149 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:34.079890 1149 webserver.cc:492] Webserver started at http://127.0.104.1:42983/ using document root <none> and password file <none>
I20251028 14:51:34.080917 1149 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:34.081070 1149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:34.081560 1149 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:34.084810 1149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "07f8e2e889374d5ab3fd4938b9ced456"
format_stamp: "Formatted at 2025-10-28 14:51:34 on dist-test-slave-0v9m"
I20251028 14:51:34.085923 1149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "07f8e2e889374d5ab3fd4938b9ced456"
format_stamp: "Formatted at 2025-10-28 14:51:34 on dist-test-slave-0v9m"
I20251028 14:51:34.094372 1149 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.003s
I20251028 14:51:34.100598 1163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:34.103176 1149 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20251028 14:51:34.103488 1149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "07f8e2e889374d5ab3fd4938b9ced456"
format_stamp: "Formatted at 2025-10-28 14:51:34 on dist-test-slave-0v9m"
I20251028 14:51:34.103777 1149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:34.149768 1149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:34.150800 1149 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:34.151074 1149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:34.154397 1149 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:34.154520 1149 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:34.154644 1149 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:34.154729 1149 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:34.203267 1149 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40787
I20251028 14:51:34.203301 1283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40787 every 8 connection(s)
I20251028 14:51:34.204794 1149 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:51:34.207401 1284 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35435
I20251028 14:51:34.207720 1284 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:34.208573 1284 heartbeater.cc:507] Master 127.0.104.62:35435 requested a full tablet report, sending...
I20251028 14:51:34.210772 1075 ts_manager.cc:194] Registered new tserver with Master: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787)
I20251028 14:51:34.212469 1075 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:52521
I20251028 14:51:34.212802 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 1149
I20251028 14:51:34.213003 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:51:34.217031 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35435
--builtin_ntp_servers=127.0.104.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:51:34.567261 1286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:34.567590 1286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:34.567699 1286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:34.576022 1286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:34.576181 1286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:34.576293 1286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:51:34.586310 1286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:34.587940 1286 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:34.589852 1286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:34.599784 1291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:34.599841 1292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:34.600224 1294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:34.600880 1286 server_base.cc:1047] running on GCE node
I20251028 14:51:34.601919 1286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:34.604007 1286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:34.605438 1286 hybrid_clock.cc:648] HybridClock initialized: now 1761663094605340 us; error 67 us; skew 500 ppm
I20251028 14:51:34.605834 1286 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:34.608721 1286 webserver.cc:492] Webserver started at http://127.0.104.2:40819/ using document root <none> and password file <none>
I20251028 14:51:34.609644 1286 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:34.609807 1286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:34.610203 1286 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:34.612967 1286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "915cee7fab2946328556aeb1aa104217"
format_stamp: "Formatted at 2025-10-28 14:51:34 on dist-test-slave-0v9m"
I20251028 14:51:34.614102 1286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "915cee7fab2946328556aeb1aa104217"
format_stamp: "Formatted at 2025-10-28 14:51:34 on dist-test-slave-0v9m"
I20251028 14:51:34.621577 1286 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20251028 14:51:34.626910 1300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:34.629210 1286 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20251028 14:51:34.629618 1286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "915cee7fab2946328556aeb1aa104217"
format_stamp: "Formatted at 2025-10-28 14:51:34 on dist-test-slave-0v9m"
I20251028 14:51:34.630030 1286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:34.658363 1286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:34.659343 1286 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:34.659657 1286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:34.663834 1286 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:34.664034 1286 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:34.664244 1286 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:34.664405 1286 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:34.683578 1075 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34982:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:51:34.729668 1286 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43371
I20251028 14:51:34.729708 1425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43371 every 8 connection(s)
I20251028 14:51:34.731395 1286 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:51:34.734194 1426 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35435
I20251028 14:51:34.734688 1426 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:34.735806 1426 heartbeater.cc:507] Master 127.0.104.62:35435 requested a full tablet report, sending...
I20251028 14:51:34.737680 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 1286
I20251028 14:51:34.737895 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:51:34.737985 1075 ts_manager.cc:194] Registered new tserver with Master: 915cee7fab2946328556aeb1aa104217 (127.0.104.2:43371)
I20251028 14:51:34.739019 1075 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:55131
I20251028 14:51:34.742053 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35435
--builtin_ntp_servers=127.0.104.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:51:35.103215 1428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:35.103560 1428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:35.103667 1428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:35.112145 1428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:35.112311 1428 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:35.112421 1428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:51:35.122725 1428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35435
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:35.124493 1428 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:35.126643 1428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:35.138484 1434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:35.138525 1436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:35.138578 1433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:35.139015 1428 server_base.cc:1047] running on GCE node
I20251028 14:51:35.139837 1428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:35.140848 1428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:35.142124 1428 hybrid_clock.cc:648] HybridClock initialized: now 1761663095142042 us; error 59 us; skew 500 ppm
I20251028 14:51:35.142472 1428 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:35.145113 1428 webserver.cc:492] Webserver started at http://127.0.104.3:45049/ using document root <none> and password file <none>
I20251028 14:51:35.145987 1428 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:35.146116 1428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:35.146502 1428 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:35.149042 1428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "bf465c0962c24ca49941bc46d5121367"
format_stamp: "Formatted at 2025-10-28 14:51:35 on dist-test-slave-0v9m"
I20251028 14:51:35.149941 1428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "bf465c0962c24ca49941bc46d5121367"
format_stamp: "Formatted at 2025-10-28 14:51:35 on dist-test-slave-0v9m"
I20251028 14:51:35.157034 1428 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20251028 14:51:35.162065 1442 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:35.164151 1428 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20251028 14:51:35.164361 1428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "bf465c0962c24ca49941bc46d5121367"
format_stamp: "Formatted at 2025-10-28 14:51:35 on dist-test-slave-0v9m"
I20251028 14:51:35.164580 1428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:35.183437 1428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:35.184347 1428 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:35.184625 1428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:35.187739 1428 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:35.187878 1428 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:35.187997 1428 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:35.188072 1428 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:35.216477 1284 heartbeater.cc:499] Master 127.0.104.62:35435 was elected leader, sending a full tablet report...
I20251028 14:51:35.246614 1428 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:33861
I20251028 14:51:35.246649 1561 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:33861 every 8 connection(s)
I20251028 14:51:35.248175 1428 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:51:35.251340 1562 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35435
I20251028 14:51:35.251796 1562 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:35.252770 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 1428
I20251028 14:51:35.252992 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:51:35.252967 1562 heartbeater.cc:507] Master 127.0.104.62:35435 requested a full tablet report, sending...
I20251028 14:51:35.255009 1074 ts_manager.cc:194] Registered new tserver with Master: bf465c0962c24ca49941bc46d5121367 (127.0.104.3:33861)
I20251028 14:51:35.256011 1074 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:34385
I20251028 14:51:35.257289 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:51:35.276372 1075 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34988:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:51:35.277506 1075 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:51:35.313526 1219 tablet_service.cc:1505] Processing CreateTablet for tablet 793dce1569ed41e3aad8035b9da60839 (DEFAULT_TABLE table=txn_write_ops_test [id=856490832ec24cfd86dcf4dd0bdc4db6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:35.313194 1496 tablet_service.cc:1505] Processing CreateTablet for tablet 7a7a7e704e324db593496fb85e9f5634 (DEFAULT_TABLE table=txn_write_ops_test [id=856490832ec24cfd86dcf4dd0bdc4db6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:35.314821 1219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793dce1569ed41e3aad8035b9da60839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.314857 1496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7a7e704e324db593496fb85e9f5634. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.312740 1497 tablet_service.cc:1505] Processing CreateTablet for tablet 793dce1569ed41e3aad8035b9da60839 (DEFAULT_TABLE table=txn_write_ops_test [id=856490832ec24cfd86dcf4dd0bdc4db6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:35.312740 1218 tablet_service.cc:1505] Processing CreateTablet for tablet 7a7a7e704e324db593496fb85e9f5634 (DEFAULT_TABLE table=txn_write_ops_test [id=856490832ec24cfd86dcf4dd0bdc4db6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:35.317039 1497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793dce1569ed41e3aad8035b9da60839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.317065 1218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7a7e704e324db593496fb85e9f5634. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.322286 1361 tablet_service.cc:1505] Processing CreateTablet for tablet 793dce1569ed41e3aad8035b9da60839 (DEFAULT_TABLE table=txn_write_ops_test [id=856490832ec24cfd86dcf4dd0bdc4db6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:35.322779 1360 tablet_service.cc:1505] Processing CreateTablet for tablet 7a7a7e704e324db593496fb85e9f5634 (DEFAULT_TABLE table=txn_write_ops_test [id=856490832ec24cfd86dcf4dd0bdc4db6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:35.324285 1361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 793dce1569ed41e3aad8035b9da60839. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.326960 1360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7a7e704e324db593496fb85e9f5634. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.352485 1579 tablet_bootstrap.cc:492] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456: Bootstrap starting.
I20251028 14:51:35.358770 1580 tablet_bootstrap.cc:492] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367: Bootstrap starting.
I20251028 14:51:35.358856 1579 tablet_bootstrap.cc:654] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.361491 1581 tablet_bootstrap.cc:492] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217: Bootstrap starting.
I20251028 14:51:35.362206 1579 log.cc:826] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:35.367321 1581 tablet_bootstrap.cc:654] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.367316 1580 tablet_bootstrap.cc:654] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.369119 1581 log.cc:826] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:35.369505 1580 log.cc:826] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:35.376963 1581 tablet_bootstrap.cc:492] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217: No bootstrap required, opened a new log
I20251028 14:51:35.377445 1581 ts_tablet_manager.cc:1403] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20251028 14:51:35.379675 1579 tablet_bootstrap.cc:492] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456: No bootstrap required, opened a new log
I20251028 14:51:35.380067 1579 ts_tablet_manager.cc:1403] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.012s
I20251028 14:51:35.386617 1581 raft_consensus.cc:359] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.387154 1581 raft_consensus.cc:385] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.387274 1581 raft_consensus.cc:740] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915cee7fab2946328556aeb1aa104217, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.387895 1579 raft_consensus.cc:359] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.388094 1581 consensus_queue.cc:260] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.388355 1579 raft_consensus.cc:385] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.388466 1579 raft_consensus.cc:740] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f8e2e889374d5ab3fd4938b9ced456, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.389334 1579 consensus_queue.cc:260] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.393491 1580 tablet_bootstrap.cc:492] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367: No bootstrap required, opened a new log
I20251028 14:51:35.394209 1580 ts_tablet_manager.cc:1403] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367: Time spent bootstrapping tablet: real 0.036s user 0.015s sys 0.012s
I20251028 14:51:35.394966 1579 ts_tablet_manager.cc:1434] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20251028 14:51:35.398440 1579 tablet_bootstrap.cc:492] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456: Bootstrap starting.
I20251028 14:51:35.400663 1579 tablet_bootstrap.cc:654] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.402304 1426 heartbeater.cc:499] Master 127.0.104.62:35435 was elected leader, sending a full tablet report...
I20251028 14:51:35.403013 1581 ts_tablet_manager.cc:1434] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217: Time spent starting tablet: real 0.025s user 0.019s sys 0.004s
I20251028 14:51:35.403688 1581 tablet_bootstrap.cc:492] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217: Bootstrap starting.
I20251028 14:51:35.403930 1580 raft_consensus.cc:359] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.404580 1580 raft_consensus.cc:385] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.404755 1580 raft_consensus.cc:740] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf465c0962c24ca49941bc46d5121367, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.405822 1580 consensus_queue.cc:260] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.406111 1581 tablet_bootstrap.cc:654] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.410522 1581 tablet_bootstrap.cc:492] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217: No bootstrap required, opened a new log
I20251028 14:51:35.410761 1581 ts_tablet_manager.cc:1403] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20251028 14:51:35.412011 1581 raft_consensus.cc:359] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.412583 1581 raft_consensus.cc:385] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.412559 1586 raft_consensus.cc:493] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:35.412706 1581 raft_consensus.cc:740] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915cee7fab2946328556aeb1aa104217, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.412892 1586 raft_consensus.cc:515] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.412981 1581 consensus_queue.cc:260] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.413694 1581 ts_tablet_manager.cc:1434] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251028 14:51:35.414753 1562 heartbeater.cc:499] Master 127.0.104.62:35435 was elected leader, sending a full tablet report...
I20251028 14:51:35.415266 1586 leader_election.cc:290] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf465c0962c24ca49941bc46d5121367 (127.0.104.3:33861), 915cee7fab2946328556aeb1aa104217 (127.0.104.2:43371)
I20251028 14:51:35.415823 1580 ts_tablet_manager.cc:1434] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367: Time spent starting tablet: real 0.021s user 0.012s sys 0.007s
I20251028 14:51:35.416527 1580 tablet_bootstrap.cc:492] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367: Bootstrap starting.
I20251028 14:51:35.418769 1580 tablet_bootstrap.cc:654] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.424237 1579 tablet_bootstrap.cc:492] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456: No bootstrap required, opened a new log
I20251028 14:51:35.424476 1579 ts_tablet_manager.cc:1403] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.003s
I20251028 14:51:35.426020 1579 raft_consensus.cc:359] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.426417 1579 raft_consensus.cc:385] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.426535 1579 raft_consensus.cc:740] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f8e2e889374d5ab3fd4938b9ced456, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.426868 1579 consensus_queue.cc:260] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.427590 1579 ts_tablet_manager.cc:1434] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251028 14:51:35.430094 1580 tablet_bootstrap.cc:492] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367: No bootstrap required, opened a new log
I20251028 14:51:35.430387 1580 ts_tablet_manager.cc:1403] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20251028 14:51:35.431442 1580 raft_consensus.cc:359] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.431768 1580 raft_consensus.cc:385] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.431905 1580 raft_consensus.cc:740] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf465c0962c24ca49941bc46d5121367, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.432283 1580 consensus_queue.cc:260] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.432950 1580 ts_tablet_manager.cc:1434] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251028 14:51:35.439042 1517 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "793dce1569ed41e3aad8035b9da60839" candidate_uuid: "07f8e2e889374d5ab3fd4938b9ced456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf465c0962c24ca49941bc46d5121367" is_pre_election: true
I20251028 14:51:35.439563 1517 raft_consensus.cc:2468] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f8e2e889374d5ab3fd4938b9ced456 in term 0.
I20251028 14:51:35.440529 1165 leader_election.cc:304] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f8e2e889374d5ab3fd4938b9ced456, bf465c0962c24ca49941bc46d5121367; no voters:
I20251028 14:51:35.440552 1381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "793dce1569ed41e3aad8035b9da60839" candidate_uuid: "07f8e2e889374d5ab3fd4938b9ced456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915cee7fab2946328556aeb1aa104217" is_pre_election: true
I20251028 14:51:35.441083 1586 raft_consensus.cc:2804] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:35.441136 1381 raft_consensus.cc:2468] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07f8e2e889374d5ab3fd4938b9ced456 in term 0.
I20251028 14:51:35.441313 1586 raft_consensus.cc:493] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:35.441440 1586 raft_consensus.cc:3060] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.443805 1586 raft_consensus.cc:515] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.444489 1586 leader_election.cc:290] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [CANDIDATE]: Term 1 election: Requested vote from peers bf465c0962c24ca49941bc46d5121367 (127.0.104.3:33861), 915cee7fab2946328556aeb1aa104217 (127.0.104.2:43371)
I20251028 14:51:35.445531 1517 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "793dce1569ed41e3aad8035b9da60839" candidate_uuid: "07f8e2e889374d5ab3fd4938b9ced456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf465c0962c24ca49941bc46d5121367"
I20251028 14:51:35.445550 1381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "793dce1569ed41e3aad8035b9da60839" candidate_uuid: "07f8e2e889374d5ab3fd4938b9ced456" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915cee7fab2946328556aeb1aa104217"
I20251028 14:51:35.445763 1381 raft_consensus.cc:3060] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.445837 1517 raft_consensus.cc:3060] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.448303 1381 raft_consensus.cc:2468] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f8e2e889374d5ab3fd4938b9ced456 in term 1.
I20251028 14:51:35.448853 1517 raft_consensus.cc:2468] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07f8e2e889374d5ab3fd4938b9ced456 in term 1.
I20251028 14:51:35.449106 1166 leader_election.cc:304] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f8e2e889374d5ab3fd4938b9ced456, 915cee7fab2946328556aeb1aa104217; no voters:
I20251028 14:51:35.449545 1586 raft_consensus.cc:2804] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:35.450071 1586 raft_consensus.cc:697] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [term 1 LEADER]: Becoming Leader. State: Replica: 07f8e2e889374d5ab3fd4938b9ced456, State: Running, Role: LEADER
I20251028 14:51:35.450664 1586 consensus_queue.cc:237] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } }
I20251028 14:51:35.456784 1075 catalog_manager.cc:5649] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 reported cstate change: term changed from 0 to 1, leader changed from <none> to 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "07f8e2e889374d5ab3fd4938b9ced456" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } health_report { overall_health: UNKNOWN } } }
W20251028 14:51:35.457612 1285 tablet.cc:2378] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:51:35.483788 1427 tablet.cc:2378] T 793dce1569ed41e3aad8035b9da60839 P 915cee7fab2946328556aeb1aa104217: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:51:35.500092 1563 tablet.cc:2378] T 793dce1569ed41e3aad8035b9da60839 P bf465c0962c24ca49941bc46d5121367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:51:35.557890 1588 raft_consensus.cc:493] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:35.558151 1588 raft_consensus.cc:515] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.559906 1588 leader_election.cc:290] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787), 915cee7fab2946328556aeb1aa104217 (127.0.104.2:43371)
I20251028 14:51:35.568027 1381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a7a7e704e324db593496fb85e9f5634" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915cee7fab2946328556aeb1aa104217" is_pre_election: true
I20251028 14:51:35.568305 1381 raft_consensus.cc:2468] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 0.
I20251028 14:51:35.568423 1239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a7a7e704e324db593496fb85e9f5634" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f8e2e889374d5ab3fd4938b9ced456" is_pre_election: true
I20251028 14:51:35.568835 1239 raft_consensus.cc:2468] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 0.
I20251028 14:51:35.568866 1445 leader_election.cc:304] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915cee7fab2946328556aeb1aa104217, bf465c0962c24ca49941bc46d5121367; no voters:
I20251028 14:51:35.569342 1588 raft_consensus.cc:2804] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:35.569479 1588 raft_consensus.cc:493] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:35.569579 1588 raft_consensus.cc:3060] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.572330 1588 raft_consensus.cc:515] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.573105 1588 leader_election.cc:290] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 election: Requested vote from peers 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787), 915cee7fab2946328556aeb1aa104217 (127.0.104.2:43371)
I20251028 14:51:35.574095 1239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a7a7e704e324db593496fb85e9f5634" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f8e2e889374d5ab3fd4938b9ced456"
I20251028 14:51:35.574074 1381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a7a7e704e324db593496fb85e9f5634" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915cee7fab2946328556aeb1aa104217"
I20251028 14:51:35.574331 1239 raft_consensus.cc:3060] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.574352 1381 raft_consensus.cc:3060] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.577693 1239 raft_consensus.cc:2468] T 7a7a7e704e324db593496fb85e9f5634 P 07f8e2e889374d5ab3fd4938b9ced456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 1.
I20251028 14:51:35.577682 1381 raft_consensus.cc:2468] T 7a7a7e704e324db593496fb85e9f5634 P 915cee7fab2946328556aeb1aa104217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 1.
I20251028 14:51:35.578284 1445 leader_election.cc:304] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 915cee7fab2946328556aeb1aa104217, bf465c0962c24ca49941bc46d5121367; no voters:
I20251028 14:51:35.578653 1588 raft_consensus.cc:2804] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:35.579336 1588 raft_consensus.cc:697] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [term 1 LEADER]: Becoming Leader. State: Replica: bf465c0962c24ca49941bc46d5121367, State: Running, Role: LEADER
I20251028 14:51:35.579962 1588 consensus_queue.cc:237] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.585624 1075 catalog_manager.cc:5649] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf465c0962c24ca49941bc46d5121367 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } health_report { overall_health: HEALTHY } } }
I20251028 14:51:35.646879 416 test_util.cc:276] Using random seed: 654813805
W20251028 14:51:35.649945 1566 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:51:35.700570 1075 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39418:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:51:35.701397 1075 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:51:35.709431 1219 tablet_service.cc:1505] Processing CreateTablet for tablet 28556c6e805144d3ad7bce547499d505 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=298fce44c0594734a919ce4fd5b067e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:51:35.709520 1360 tablet_service.cc:1505] Processing CreateTablet for tablet 28556c6e805144d3ad7bce547499d505 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=298fce44c0594734a919ce4fd5b067e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:51:35.710299 1219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28556c6e805144d3ad7bce547499d505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.710309 1360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28556c6e805144d3ad7bce547499d505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.713747 1497 tablet_service.cc:1505] Processing CreateTablet for tablet 28556c6e805144d3ad7bce547499d505 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=298fce44c0594734a919ce4fd5b067e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:51:35.714669 1497 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28556c6e805144d3ad7bce547499d505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:35.716872 1581 tablet_bootstrap.cc:492] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: Bootstrap starting.
I20251028 14:51:35.718755 1579 tablet_bootstrap.cc:492] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: Bootstrap starting.
I20251028 14:51:35.718750 1581 tablet_bootstrap.cc:654] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.720554 1579 tablet_bootstrap.cc:654] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.722450 1581 tablet_bootstrap.cc:492] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: No bootstrap required, opened a new log
I20251028 14:51:35.722646 1581 ts_tablet_manager.cc:1403] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251028 14:51:35.722795 1580 tablet_bootstrap.cc:492] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: Bootstrap starting.
I20251028 14:51:35.723497 1581 raft_consensus.cc:359] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.723774 1581 raft_consensus.cc:385] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.723874 1581 raft_consensus.cc:740] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 915cee7fab2946328556aeb1aa104217, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.724206 1581 consensus_queue.cc:260] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.724624 1585 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:35.724846 1580 tablet_bootstrap.cc:654] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:35.724959 1585 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:35.725418 1581 ts_tablet_manager.cc:1434] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251028 14:51:35.735776 1579 tablet_bootstrap.cc:492] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: No bootstrap required, opened a new log
I20251028 14:51:35.736003 1579 ts_tablet_manager.cc:1403] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20251028 14:51:35.736910 1579 raft_consensus.cc:359] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.737166 1579 raft_consensus.cc:385] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.737284 1579 raft_consensus.cc:740] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07f8e2e889374d5ab3fd4938b9ced456, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.737592 1579 consensus_queue.cc:260] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.738142 1586 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:35.738405 1579 ts_tablet_manager.cc:1434] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251028 14:51:35.738467 1586 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:35.739414 1580 tablet_bootstrap.cc:492] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: No bootstrap required, opened a new log
I20251028 14:51:35.739599 1580 ts_tablet_manager.cc:1403] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20251028 14:51:35.740471 1580 raft_consensus.cc:359] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.740700 1580 raft_consensus.cc:385] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:35.740765 1580 raft_consensus.cc:740] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf465c0962c24ca49941bc46d5121367, State: Initialized, Role: FOLLOWER
I20251028 14:51:35.741037 1580 consensus_queue.cc:260] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.741667 1580 ts_tablet_manager.cc:1434] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251028 14:51:35.741636 1588 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:35.741951 1588 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:35.832638 1586 consensus_queue.cc:1048] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:35.842808 1593 consensus_queue.cc:1048] T 793dce1569ed41e3aad8035b9da60839 P 07f8e2e889374d5ab3fd4938b9ced456 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:35.878767 1588 raft_consensus.cc:493] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:35.879036 1588 raft_consensus.cc:515] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.879747 1588 leader_election.cc:290] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787), 915cee7fab2946328556aeb1aa104217 (127.0.104.2:43371)
I20251028 14:51:35.880214 1239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28556c6e805144d3ad7bce547499d505" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f8e2e889374d5ab3fd4938b9ced456" is_pre_election: true
I20251028 14:51:35.880347 1381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28556c6e805144d3ad7bce547499d505" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915cee7fab2946328556aeb1aa104217" is_pre_election: true
I20251028 14:51:35.880446 1239 raft_consensus.cc:2468] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 0.
I20251028 14:51:35.880563 1381 raft_consensus.cc:2468] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 0.
I20251028 14:51:35.880859 1446 leader_election.cc:304] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f8e2e889374d5ab3fd4938b9ced456, bf465c0962c24ca49941bc46d5121367; no voters:
I20251028 14:51:35.881191 1588 raft_consensus.cc:2804] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:35.881338 1588 raft_consensus.cc:493] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:35.881445 1588 raft_consensus.cc:3060] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.883459 1588 raft_consensus.cc:515] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.884138 1588 leader_election.cc:290] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 election: Requested vote from peers 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787), 915cee7fab2946328556aeb1aa104217 (127.0.104.2:43371)
I20251028 14:51:35.884712 1381 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28556c6e805144d3ad7bce547499d505" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "915cee7fab2946328556aeb1aa104217"
I20251028 14:51:35.884732 1239 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28556c6e805144d3ad7bce547499d505" candidate_uuid: "bf465c0962c24ca49941bc46d5121367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07f8e2e889374d5ab3fd4938b9ced456"
I20251028 14:51:35.884915 1381 raft_consensus.cc:3060] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.884930 1239 raft_consensus.cc:3060] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:35.887038 1239 raft_consensus.cc:2468] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 1.
I20251028 14:51:35.887238 1381 raft_consensus.cc:2468] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf465c0962c24ca49941bc46d5121367 in term 1.
I20251028 14:51:35.887568 1446 leader_election.cc:304] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07f8e2e889374d5ab3fd4938b9ced456, bf465c0962c24ca49941bc46d5121367; no voters:
I20251028 14:51:35.888098 1588 raft_consensus.cc:2804] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:35.888551 1588 raft_consensus.cc:697] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [term 1 LEADER]: Becoming Leader. State: Replica: bf465c0962c24ca49941bc46d5121367, State: Running, Role: LEADER
I20251028 14:51:35.888959 1588 consensus_queue.cc:237] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } }
I20251028 14:51:35.890967 1596 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: TxnStatusTablet state changed. Reason: New leader bf465c0962c24ca49941bc46d5121367. Latest consensus state: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:35.891244 1596 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:35.892047 1622 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:51:35.892455 1622 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:51:35.892823 1075 catalog_manager.cc:5649] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf465c0962c24ca49941bc46d5121367 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } health_report { overall_health: HEALTHY } } }
I20251028 14:51:36.060596 1381 raft_consensus.cc:1275] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217 [term 1 FOLLOWER]: Refusing update from remote peer bf465c0962c24ca49941bc46d5121367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:36.061048 1239 raft_consensus.cc:1275] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456 [term 1 FOLLOWER]: Refusing update from remote peer bf465c0962c24ca49941bc46d5121367: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:36.061416 1588 consensus_queue.cc:1048] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:36.061872 1596 consensus_queue.cc:1048] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:36.071286 1593 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: TxnStatusTablet state changed. Reason: New leader bf465c0962c24ca49941bc46d5121367. Latest consensus state: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:36.071523 1593 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:36.073410 1620 mvcc.cc:204] Tried to move back new op lower bound from 7215772041453322240 to 7215772040765435904. Current Snapshot: MvccSnapshot[applied={T|T < 7215772041453322240}]
I20251028 14:51:36.080837 1585 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: TxnStatusTablet state changed. Reason: New leader bf465c0962c24ca49941bc46d5121367. Latest consensus state: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:36.081126 1585 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:36.084074 1588 consensus_queue.cc:1048] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:36.088335 1618 mvcc.cc:204] Tried to move back new op lower bound from 7215772041453322240 to 7215772040765435904. Current Snapshot: MvccSnapshot[applied={T|T < 7215772041453322240}]
I20251028 14:51:36.097011 1588 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:36.097390 1588 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:36.099460 1593 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:36.099699 1593 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P 07f8e2e889374d5ab3fd4938b9ced456: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:36.110661 1588 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:36.111639 1634 consensus_queue.cc:1048] T 7a7a7e704e324db593496fb85e9f5634 P bf465c0962c24ca49941bc46d5121367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:36.112255 1588 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P bf465c0962c24ca49941bc46d5121367: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:36.129832 1585 tablet_replica.cc:442] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bf465c0962c24ca49941bc46d5121367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07f8e2e889374d5ab3fd4938b9ced456" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40787 } } peers { permanent_uuid: "915cee7fab2946328556aeb1aa104217" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43371 } } peers { permanent_uuid: "bf465c0962c24ca49941bc46d5121367" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33861 } } }
I20251028 14:51:36.130062 1585 tablet_replica.cc:445] T 28556c6e805144d3ad7bce547499d505 P 915cee7fab2946328556aeb1aa104217: This TxnStatusTablet replica's current role is: FOLLOWER
W20251028 14:51:36.136873 1566 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.104.62:35435, user_credentials={real_user=slave}} blocked reactor thread for 53354.8us
W20251028 14:51:36.279866 1201 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.104.62:35435, user_credentials={real_user=slave}} blocked reactor thread for 92796.1us
I20251028 14:51:36.368043 1601 txn_write_ops-itest.cc:454] Txn 1 wrote row 7: OK
I20251028 14:51:36.428838 1600 txn_write_ops-itest.cc:454] Txn 7 wrote row 12: OK
W20251028 14:51:37.368446 1656 lock_manager.cc:494] Txn 3 has not acquired the partition lock after 1000ms
I20251028 14:51:37.369344 1663 ts_tablet_manager.cc:2121] Sending abort request for transaction 3
W20251028 14:51:37.369225 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:37.369925 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=3, attempt_no=0}) took 1192 ms. Trace:
I20251028 14:51:37.370047 1656 rpcz_store.cc:276] 1028 14:51:36.177705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.178064 (+ 359us) service_pool.cc:225] Handling call
1028 14:51:37.369856 (+1191792us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:36.359135 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:36.359256 (+ 121us) participant_op.cc:199] PREPARE: finished
1028 14:51:36.359288 (+ 32us) participant_op.cc:212] START. Timestamp: 1761663096359246
1028 14:51:36.359291 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:36.359407 (+ 116us) log.cc:844] Serialized 65 byte log entry
1028 14:51:36.364602 (+ 5195us) op_driver.cc:464] REPLICATION: finished
1028 14:51:36.366123 (+ 1521us) participant_op.cc:265] APPLY: starting
1028 14:51:36.366821 (+ 698us) participant_op.cc:270] APPLY: finished
1028 14:51:36.366872 (+ 51us) log.cc:844] Serialized 14 byte log entry
1028 14:51:36.366914 (+ 42us) participant_op.cc:94] Released txn lock
1028 14:51:36.366924 (+ 10us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:36.367468 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:36.367590 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:36.367594 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:51:36.367596 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:36.367679 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:37.368674 (+1000995us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:51:37.368693 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:37.368702 (+ 9us) write_op.cc:454] Released schema lock
1028 14:51:37.368714 (+ 12us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":100605,"threads_started":1,"txn-participant-registration.queue_time_us":110322,"txn-participant-registration.run_cpu_time_us":2619,"txn-participant-registration.run_wall_time_us":69481,"child_traces":[["op",{"apply.queue_time_us":1484,"apply.run_cpu_time_us":1202,"apply.run_wall_time_us":1201,"mutex_wait_us":2507,"prepare.queue_time_us":3144,"prepare.run_cpu_time_us":340,"prepare.run_wall_time_us":339,"raft.queue_time_us":243,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":5294,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":176,"thread_start_us":271,"threads_started":1,"txn-participant-registration.queue_time_us":473}]]}]]}
W20251028 14:51:37.374826 1567 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:37.375545 1602 txn_write_ops-itest.cc:454] Txn 3 wrote row 3: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:37.375685 1602 txn_write_ops-itest.cc:465] Txn 3 wrote row 3: Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:51:37.431208 1657 lock_manager.cc:494] Txn 4 has not acquired the partition lock after 1000ms
W20251028 14:51:37.431566 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:37.432186 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=4, attempt_no=0}) took 1187 ms. Trace:
I20251028 14:51:37.432325 1657 rpcz_store.cc:276] 1028 14:51:36.244460 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.244636 (+ 176us) service_pool.cc:225] Handling call
1028 14:51:37.432130 (+1187494us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:36.417672 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:36.417731 (+ 59us) participant_op.cc:199] PREPARE: finished
1028 14:51:36.417779 (+ 48us) participant_op.cc:212] START. Timestamp: 1761663096417727
1028 14:51:36.417782 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:36.417983 (+ 201us) log.cc:844] Serialized 65 byte log entry
1028 14:51:36.426775 (+ 8792us) op_driver.cc:464] REPLICATION: finished
1028 14:51:36.428793 (+ 2018us) participant_op.cc:265] APPLY: starting
1028 14:51:36.429688 (+ 895us) participant_op.cc:270] APPLY: finished
1028 14:51:36.429749 (+ 61us) log.cc:844] Serialized 14 byte log entry
1028 14:51:36.429788 (+ 39us) participant_op.cc:94] Released txn lock
1028 14:51:36.429797 (+ 9us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:36.430300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:36.430405 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:36.430408 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:36.430410 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:36.430509 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:37.431458 (+1000949us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:37.431480 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:37.431490 (+ 10us) write_op.cc:454] Released schema lock
1028 14:51:37.431507 (+ 17us) write_op.cc:335] FINISH: Op aborted
Metrics: {"dns-resolver.queue_time_us":21966,"dns-resolver.run_cpu_time_us":577,"dns-resolver.run_wall_time_us":575,"dns_us":137,"thread_start_us":50534,"threads_started":1,"txn-participant-registration.queue_time_us":46318,"txn-participant-registration.run_cpu_time_us":2220,"txn-participant-registration.run_wall_time_us":114001,"child_traces":[["op",{"apply.queue_time_us":1912,"apply.run_cpu_time_us":1316,"apply.run_wall_time_us":1409,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":496,"prepare.run_wall_time_us":5576,"replication_time_us":8974,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":139}]]}]]}
W20251028 14:51:37.439356 1476 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.0.104.62:35435, user_credentials={real_user=slave}} blocked reactor thread for 44105.8us
I20251028 14:51:37.450284 1671 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=9, attempt_no=0}) took 1020 ms. Trace:
I20251028 14:51:37.450464 1671 rpcz_store.cc:276] 1028 14:51:36.430001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.430068 (+ 67us) service_pool.cc:225] Handling call
1028 14:51:37.450207 (+1020139us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:37.434447 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:37.434574 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:37.434578 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:51:37.434580 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:37.434664 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:37.434676 (+ 12us) write_op.cc:641] Partition lock acquired for write op
1028 14:51:37.434681 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:51:37.434738 (+ 57us) tablet.cc:699] Row locks acquired
1028 14:51:37.434741 (+ 3us) write_op.cc:260] PREPARE: finished
1028 14:51:37.434798 (+ 57us) write_op.cc:270] Start()
1028 14:51:37.434837 (+ 39us) write_op.cc:276] Timestamp: P: 1761663097434757 usec, L: 0
1028 14:51:37.434840 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:37.435115 (+ 275us) log.cc:844] Serialized 302 byte log entry
1028 14:51:37.449497 (+ 14382us) op_driver.cc:464] REPLICATION: finished
1028 14:51:37.449590 (+ 93us) write_op.cc:301] APPLY: starting
1028 14:51:37.449618 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
1028 14:51:37.449662 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1028 14:51:37.449697 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:51:37.449773 (+ 76us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:51:37.449808 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1028 14:51:37.449815 (+ 7us) write_op.cc:312] APPLY: finished
1028 14:51:37.449922 (+ 107us) log.cc:844] Serialized 24 byte log entry
1028 14:51:37.450048 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:37.450086 (+ 38us) write_op.cc:454] Released schema lock
1028 14:51:37.450093 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"mutex_wait_us":3,"num_ops":1,"prepare.queue_time_us":1003823,"prepare.run_cpu_time_us":5991,"prepare.run_wall_time_us":11185,"raft.queue_time_us":343,"raft.run_cpu_time_us":860,"raft.run_wall_time_us":1325,"replication_time_us":14624,"spinlock_wait_cycles":1317760,"thread_start_us":10274,"threads_started":1,"wal-append.queue_time_us":10144}]]}
I20251028 14:51:37.451805 1600 txn_write_ops-itest.cc:454] Txn 7 wrote row 10: OK
W20251028 14:51:38.374832 1656 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:51:38.375160 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:38.375499 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=0, attempt_no=0}) took 2216 ms. Trace:
I20251028 14:51:38.375598 1656 rpcz_store.cc:276] 1028 14:51:36.158617 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.158935 (+ 318us) service_pool.cc:225] Handling call
1028 14:51:38.375432 (+2216497us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:36.364950 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:36.364992 (+ 42us) participant_op.cc:199] PREPARE: finished
1028 14:51:36.365021 (+ 29us) participant_op.cc:212] START. Timestamp: 1761663096364979
1028 14:51:36.365024 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:36.365166 (+ 142us) log.cc:844] Serialized 65 byte log entry
1028 14:51:36.374542 (+ 9376us) op_driver.cc:464] REPLICATION: finished
1028 14:51:36.374657 (+ 115us) participant_op.cc:265] APPLY: starting
1028 14:51:36.374951 (+ 294us) participant_op.cc:270] APPLY: finished
1028 14:51:36.374986 (+ 35us) log.cc:844] Serialized 14 byte log entry
1028 14:51:36.375025 (+ 39us) participant_op.cc:94] Released txn lock
1028 14:51:36.375033 (+ 8us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:37.373901 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:37.374043 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:37.374046 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:37.374048 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:37.374136 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:38.375085 (+1000949us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:38.375109 (+ 24us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:38.375117 (+ 8us) write_op.cc:454] Released schema lock
1028 14:51:38.375129 (+ 12us) write_op.cc:335] FINISH: Op aborted
Metrics: {"dns-resolver.queue_time_us":24202,"dns-resolver.run_cpu_time_us":571,"dns-resolver.run_wall_time_us":571,"dns_us":121,"spinlock_wait_cycles":14208,"thread_start_us":141290,"threads_started":1,"txn-participant-registration.queue_time_us":100406,"txn-participant-registration.run_cpu_time_us":1109,"txn-participant-registration.run_wall_time_us":98469,"child_traces":[["op",{"apply.queue_time_us":52,"apply.run_cpu_time_us":665,"apply.run_wall_time_us":664,"prepare.queue_time_us":494,"prepare.run_cpu_time_us":263,"prepare.run_wall_time_us":262,"replication_time_us":9486,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":998613}]]}]]}
W20251028 14:51:38.456384 1657 lock_manager.cc:494] Txn 4 has not acquired the partition lock after 1000ms
W20251028 14:51:38.456629 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:38.456954 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=4, attempt_no=1}) took 1017 ms. Trace:
I20251028 14:51:38.457041 1657 rpcz_store.cc:276] 1028 14:51:37.439599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:37.439675 (+ 76us) service_pool.cc:225] Handling call
1028 14:51:38.456879 (+1017204us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:37.446666 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:37.455666 (+ 9000us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:37.455675 (+ 9us) write_op.cc:435] Acquired schema lock
1028 14:51:37.455678 (+ 3us) tablet.cc:660] Decoding operations
1028 14:51:37.455801 (+ 123us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:38.456574 (+1000773us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:38.456588 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:38.456594 (+ 6us) write_op.cc:454] Released schema lock
1028 14:51:38.456604 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":6374}]]}
W20251028 14:51:39.378674 1656 lock_manager.cc:494] Txn 6 has not acquired the partition lock after 1000ms
W20251028 14:51:39.379279 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:39.379437 1693 ts_tablet_manager.cc:2121] Sending abort request for transaction 6
I20251028 14:51:39.379624 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=6, attempt_no=0}) took 3201 ms. Trace:
I20251028 14:51:39.379706 1656 rpcz_store.cc:276] 1028 14:51:36.178269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.178319 (+ 50us) service_pool.cc:225] Handling call
1028 14:51:39.379561 (+3201242us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:37.373042 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:37.373091 (+ 49us) participant_op.cc:199] PREPARE: finished
1028 14:51:37.373132 (+ 41us) participant_op.cc:212] START. Timestamp: 1761663097373065
1028 14:51:37.373135 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:37.373372 (+ 237us) log.cc:844] Serialized 65 byte log entry
1028 14:51:37.398722 (+ 25350us) op_driver.cc:464] REPLICATION: finished
1028 14:51:37.439247 (+ 40525us) participant_op.cc:265] APPLY: starting
1028 14:51:37.439587 (+ 340us) participant_op.cc:270] APPLY: finished
1028 14:51:37.439651 (+ 64us) log.cc:844] Serialized 14 byte log entry
1028 14:51:37.439700 (+ 49us) participant_op.cc:94] Released txn lock
1028 14:51:37.439710 (+ 10us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:38.377603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:38.377809 (+ 206us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:38.377813 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:51:38.377815 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:38.377935 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:39.378865 (+1000930us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:51:39.378878 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:39.378884 (+ 6us) write_op.cc:454] Released schema lock
1028 14:51:39.378894 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":100463,"threads_started":1,"txn-participant-registration.queue_time_us":128298,"txn-participant-registration.run_cpu_time_us":1251,"txn-participant-registration.run_wall_time_us":66870,"child_traces":[["op",{"apply.queue_time_us":40433,"apply.run_cpu_time_us":981,"apply.run_wall_time_us":980,"prepare.queue_time_us":999151,"prepare.run_cpu_time_us":433,"prepare.run_wall_time_us":787,"raft.queue_time_us":41,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":291,"replication_time_us":25539,"thread_start_us":42028,"threads_started":1,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":937542,"thread_start_us":272,"threads_started":1,"txn-participant-registration.queue_time_us":431}]]}]]}
W20251028 14:51:39.380708 1567 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:51:39.380694 1656 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251028 14:51:39.381063 1604 txn_write_ops-itest.cc:454] Txn 6 wrote row 13: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:39.381140 1604 txn_write_ops-itest.cc:465] Txn 6 wrote row 13: Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:39.399818 1699 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:40817 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=3, attempt_no=0}) took 1945 ms. Trace:
I20251028 14:51:39.399977 1699 rpcz_store.cc:276] 1028 14:51:37.454326 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:37.454387 (+ 61us) service_pool.cc:225] Handling call
1028 14:51:39.399747 (+1945360us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:39.380562 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:39.380972 (+ 410us) participant_op.cc:199] PREPARE: finished
1028 14:51:39.381027 (+ 55us) participant_op.cc:212] START. Timestamp: 1761663099380956
1028 14:51:39.381030 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:39.381225 (+ 195us) log.cc:844] Serialized 65 byte log entry
1028 14:51:39.384229 (+ 3004us) op_driver.cc:464] REPLICATION: finished
1028 14:51:39.398812 (+ 14583us) participant_op.cc:265] APPLY: starting
1028 14:51:39.399293 (+ 481us) participant_op.cc:270] APPLY: finished
1028 14:51:39.399365 (+ 72us) log.cc:844] Serialized 14 byte log entry
1028 14:51:39.399621 (+ 256us) participant_op.cc:94] Released txn lock
1028 14:51:39.399663 (+ 42us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":14514,"prepare.queue_time_us":1925726,"prepare.run_cpu_time_us":1249,"prepare.run_wall_time_us":1247,"raft.queue_time_us":558,"raft.run_cpu_time_us":679,"raft.run_wall_time_us":1331,"replication_time_us":3166,"thread_start_us":14887,"threads_started":2}]]}
W20251028 14:51:39.458547 1657 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20251028 14:51:39.458799 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:39.459086 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=8, attempt_no=0}) took 3088 ms. Trace:
I20251028 14:51:39.459162 1657 rpcz_store.cc:276] 1028 14:51:36.370726 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.370805 (+ 79us) service_pool.cc:225] Handling call
1028 14:51:39.459034 (+3088229us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:37.445659 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:37.445717 (+ 58us) participant_op.cc:199] PREPARE: finished
1028 14:51:37.445766 (+ 49us) participant_op.cc:212] START. Timestamp: 1761663097445718
1028 14:51:37.445769 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:37.446531 (+ 762us) log.cc:844] Serialized 65 byte log entry
1028 14:51:37.454292 (+ 7761us) op_driver.cc:464] REPLICATION: finished
1028 14:51:37.454918 (+ 626us) participant_op.cc:265] APPLY: starting
1028 14:51:37.455201 (+ 283us) participant_op.cc:270] APPLY: finished
1028 14:51:37.455230 (+ 29us) log.cc:844] Serialized 14 byte log entry
1028 14:51:37.455263 (+ 33us) participant_op.cc:94] Released txn lock
1028 14:51:37.455271 (+ 8us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:38.457707 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:38.457833 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:38.457835 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:51:38.457836 (+ 1us) tablet.cc:660] Decoding operations
1028 14:51:38.457911 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:39.458745 (+1000834us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:39.458761 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:39.458768 (+ 7us) write_op.cc:454] Released schema lock
1028 14:51:39.458778 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":21125,"threads_started":1,"txn-participant-registration.queue_time_us":27753,"txn-participant-registration.run_cpu_time_us":975,"txn-participant-registration.run_wall_time_us":46528,"child_traces":[["op",{"apply.queue_time_us":562,"apply.run_cpu_time_us":628,"apply.run_wall_time_us":628,"prepare.queue_time_us":998912,"prepare.run_cpu_time_us":496,"prepare.run_wall_time_us":962,"replication_time_us":8493,"spinlock_wait_cycles":1228160,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":1002193}]]}]]}
I20251028 14:51:39.464684 1705 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:43983 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=4, attempt_no=0}) took 1999 ms. Trace:
I20251028 14:51:39.464839 1705 rpcz_store.cc:276] 1028 14:51:37.465282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:37.465368 (+ 86us) service_pool.cc:225] Handling call
1028 14:51:39.464620 (+1999252us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:39.459922 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:39.460023 (+ 101us) participant_op.cc:199] PREPARE: finished
1028 14:51:39.460080 (+ 57us) participant_op.cc:212] START. Timestamp: 1761663099460024
1028 14:51:39.460082 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:39.460236 (+ 154us) log.cc:844] Serialized 65 byte log entry
1028 14:51:39.464203 (+ 3967us) op_driver.cc:464] REPLICATION: finished
1028 14:51:39.464291 (+ 88us) participant_op.cc:265] APPLY: starting
1028 14:51:39.464434 (+ 143us) participant_op.cc:270] APPLY: finished
1028 14:51:39.464489 (+ 55us) log.cc:844] Serialized 14 byte log entry
1028 14:51:39.464530 (+ 41us) participant_op.cc:94] Released txn lock
1028 14:51:39.464538 (+ 8us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":27,"mutex_wait_us":26,"prepare.queue_time_us":1994170,"prepare.run_cpu_time_us":702,"prepare.run_wall_time_us":700,"raft.queue_time_us":103,"raft.run_cpu_time_us":805,"raft.run_wall_time_us":805,"replication_time_us":4087,"spinlock_wait_cycles":212992,"thread_start_us":238,"threads_started":1,"wal-append.queue_time_us":377}]]}
W20251028 14:51:40.382901 1656 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:51:40.383265 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:40.383543 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=0, attempt_no=1}) took 2003 ms. Trace:
I20251028 14:51:40.383644 1656 rpcz_store.cc:276] 1028 14:51:38.379939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:38.380014 (+ 75us) service_pool.cc:225] Handling call
1028 14:51:40.383506 (+2003492us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:39.381863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:39.382020 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:39.382024 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:51:39.382027 (+ 3us) tablet.cc:660] Decoding operations
1028 14:51:39.382129 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:40.383158 (+1001029us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:40.383176 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:40.383184 (+ 8us) write_op.cc:454] Released schema lock
1028 14:51:40.383195 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":1001244}]]}
W20251028 14:51:40.461697 1657 lock_manager.cc:494] Txn 4 has not acquired the partition lock after 1000ms
W20251028 14:51:40.462044 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:40.462497 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=4, attempt_no=2}) took 1998 ms. Trace:
I20251028 14:51:40.462590 1657 rpcz_store.cc:276] 1028 14:51:38.464011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:38.464162 (+ 151us) service_pool.cc:225] Handling call
1028 14:51:40.462428 (+1998266us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:39.460669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:39.460809 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:39.460812 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:39.460814 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:39.461009 (+ 195us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:40.461964 (+1000955us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:40.461988 (+ 24us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:40.461998 (+ 10us) write_op.cc:454] Released schema lock
1028 14:51:40.462012 (+ 14us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":995914}]]}
W20251028 14:51:41.385202 1656 lock_manager.cc:494] Txn 5 has not acquired the partition lock after 1000ms
W20251028 14:51:41.385871 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:41.385998 1715 ts_tablet_manager.cc:2121] Sending abort request for transaction 5
I20251028 14:51:41.386190 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=5, attempt_no=0}) took 5220 ms. Trace:
I20251028 14:51:41.386332 1656 rpcz_store.cc:276] 1028 14:51:36.165736 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.165789 (+ 53us) service_pool.cc:225] Handling call
1028 14:51:41.386145 (+5220356us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:38.376425 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:38.376471 (+ 46us) participant_op.cc:199] PREPARE: finished
1028 14:51:38.376517 (+ 46us) participant_op.cc:212] START. Timestamp: 1761663098376436
1028 14:51:38.376519 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:38.376726 (+ 207us) log.cc:844] Serialized 65 byte log entry
1028 14:51:38.380633 (+ 3907us) op_driver.cc:464] REPLICATION: finished
1028 14:51:38.381123 (+ 490us) participant_op.cc:265] APPLY: starting
1028 14:51:38.381480 (+ 357us) participant_op.cc:270] APPLY: finished
1028 14:51:38.381535 (+ 55us) log.cc:844] Serialized 14 byte log entry
1028 14:51:38.381813 (+ 278us) participant_op.cc:94] Released txn lock
1028 14:51:38.381823 (+ 10us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:40.384230 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:40.384377 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:40.384380 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:40.384382 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:40.384472 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:41.385433 (+1000961us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:51:41.385448 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:41.385454 (+ 6us) write_op.cc:454] Released schema lock
1028 14:51:41.385468 (+ 14us) write_op.cc:335] FINISH: Op aborted
Metrics: {"dns-resolver.queue_time_us":23980,"dns-resolver.run_cpu_time_us":204,"dns-resolver.run_wall_time_us":203,"dns_us":67,"thread_start_us":122341,"threads_started":1,"txn-participant-registration.queue_time_us":100711,"txn-participant-registration.run_cpu_time_us":881,"txn-participant-registration.run_wall_time_us":110729,"child_traces":[["op",{"apply.queue_time_us":410,"apply.run_cpu_time_us":749,"apply.run_wall_time_us":976,"prepare.queue_time_us":1998667,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":739,"raft.queue_time_us":473,"raft.run_cpu_time_us":710,"raft.run_wall_time_us":908,"replication_time_us":4081,"spinlock_wait_cycles":10496,"thread_start_us":588,"threads_started":2,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":2002152,"thread_start_us":259,"threads_started":1,"txn-participant-registration.queue_time_us":401}]]}]]}
W20251028 14:51:41.387619 1567 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:41.387929 1605 txn_write_ops-itest.cc:454] Txn 5 wrote row 16: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:41.388005 1605 txn_write_ops-itest.cc:465] Txn 5 wrote row 16: Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:51:41.464262 1657 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20251028 14:51:41.464689 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:41.465124 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=8, attempt_no=1}) took 2000 ms. Trace:
I20251028 14:51:41.465219 1657 rpcz_store.cc:276] 1028 14:51:39.464911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:39.464979 (+ 68us) service_pool.cc:225] Handling call
1028 14:51:41.465006 (+2000027us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:40.463413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:40.463531 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:40.463533 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:51:40.463534 (+ 1us) tablet.cc:660] Decoding operations
1028 14:51:40.463608 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:41.464534 (+1000926us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:41.464602 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:41.464618 (+ 16us) write_op.cc:454] Released schema lock
1028 14:51:41.464636 (+ 18us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":997826}]]}
I20251028 14:51:41.471170 1723 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:43983 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=6, attempt_no=0}) took 1999 ms. Trace:
I20251028 14:51:41.471369 1723 rpcz_store.cc:276] 1028 14:51:39.471357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:39.471505 (+ 148us) service_pool.cc:225] Handling call
1028 14:51:41.471091 (+1999586us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:41.466150 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:41.466246 (+ 96us) participant_op.cc:199] PREPARE: finished
1028 14:51:41.466265 (+ 19us) participant_op.cc:212] START. Timestamp: 1761663101466212
1028 14:51:41.466267 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:41.466445 (+ 178us) log.cc:844] Serialized 75 byte log entry
1028 14:51:41.470370 (+ 3925us) op_driver.cc:464] REPLICATION: finished
1028 14:51:41.470502 (+ 132us) participant_op.cc:265] APPLY: starting
1028 14:51:41.470905 (+ 403us) participant_op.cc:270] APPLY: finished
1028 14:51:41.470964 (+ 59us) log.cc:844] Serialized 14 byte log entry
1028 14:51:41.471007 (+ 43us) participant_op.cc:94] Released txn lock
1028 14:51:41.471067 (+ 60us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":79,"prepare.queue_time_us":1994290,"prepare.run_cpu_time_us":719,"prepare.run_wall_time_us":719,"raft.queue_time_us":155,"raft.run_cpu_time_us":731,"raft.run_wall_time_us":815,"replication_time_us":4055,"spinlock_wait_cycles":16640,"thread_start_us":290,"threads_started":1,"wal-append.queue_time_us":418}]]}
I20251028 14:51:41.476308 1723 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=4, attempt_no=3}) took 1008 ms. Trace:
I20251028 14:51:41.476496 1723 rpcz_store.cc:276] 1028 14:51:40.467393 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:40.467537 (+ 144us) service_pool.cc:225] Handling call
1028 14:51:41.476240 (+1008703us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:41.466944 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:41.467098 (+ 154us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:41.467102 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:51:41.467104 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:41.467190 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:41.470926 (+ 3736us) write_op.cc:641] Partition lock acquired for write op
1028 14:51:41.470943 (+ 17us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:51:41.471023 (+ 80us) tablet.cc:699] Row locks acquired
1028 14:51:41.471026 (+ 3us) write_op.cc:260] PREPARE: finished
1028 14:51:41.471113 (+ 87us) write_op.cc:270] Start()
1028 14:51:41.471151 (+ 38us) write_op.cc:276] Timestamp: P: 1761663101471058 usec, L: 0
1028 14:51:41.471155 (+ 4us) op_driver.cc:348] REPLICATION: starting
1028 14:51:41.471370 (+ 215us) log.cc:844] Serialized 293 byte log entry
1028 14:51:41.475540 (+ 4170us) op_driver.cc:464] REPLICATION: finished
1028 14:51:41.475622 (+ 82us) write_op.cc:301] APPLY: starting
1028 14:51:41.475646 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
1028 14:51:41.475716 (+ 70us) tablet.cc:1368] finished BulkCheckPresence
1028 14:51:41.475750 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:51:41.475850 (+ 100us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:51:41.475885 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1028 14:51:41.475892 (+ 7us) write_op.cc:312] APPLY: finished
1028 14:51:41.475983 (+ 91us) log.cc:844] Serialized 24 byte log entry
1028 14:51:41.476091 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:41.476112 (+ 21us) write_op.cc:454] Released schema lock
1028 14:51:41.476120 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":35,"mutex_wait_us":832,"num_ops":1,"prepare.queue_time_us":998891,"prepare.run_cpu_time_us":2253,"prepare.run_wall_time_us":6822,"raft.queue_time_us":515,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":110,"replication_time_us":4357,"spinlock_wait_cycles":749312,"thread_start_us":1126,"threads_started":1}]]}
I20251028 14:51:41.477603 1606 txn_write_ops-itest.cc:454] Txn 4 wrote row 11: OK
W20251028 14:51:42.388137 1656 lock_manager.cc:494] Txn 2 has not acquired the partition lock after 1000ms
W20251028 14:51:42.388723 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:42.388835 1741 ts_tablet_manager.cc:2121] Sending abort request for transaction 2
I20251028 14:51:42.389036 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=2, attempt_no=0}) took 6230 ms. Trace:
I20251028 14:51:42.389147 1656 rpcz_store.cc:276] 1028 14:51:36.158875 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:36.161312 (+ 2437us) service_pool.cc:225] Handling call
1028 14:51:42.388966 (+6227654us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:38.377206 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:38.377239 (+ 33us) participant_op.cc:199] PREPARE: finished
1028 14:51:38.377321 (+ 82us) participant_op.cc:212] START. Timestamp: 1761663098377238
1028 14:51:38.377324 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:38.377508 (+ 184us) log.cc:844] Serialized 65 byte log entry
1028 14:51:38.383390 (+ 5882us) op_driver.cc:464] REPLICATION: finished
1028 14:51:38.383485 (+ 95us) participant_op.cc:265] APPLY: starting
1028 14:51:38.383716 (+ 231us) participant_op.cc:270] APPLY: finished
1028 14:51:38.383757 (+ 41us) log.cc:844] Serialized 14 byte log entry
1028 14:51:38.383960 (+ 203us) participant_op.cc:94] Released txn lock
1028 14:51:38.383967 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:41.387300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:41.387442 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:41.387445 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:41.387447 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:41.387537 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:42.388306 (+1000769us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:51:42.388316 (+ 10us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:42.388321 (+ 5us) write_op.cc:454] Released schema lock
1028 14:51:42.388328 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"dns-resolver.queue_time_us":23543,"dns-resolver.run_cpu_time_us":119,"dns-resolver.run_wall_time_us":119,"dns_us":35,"thread_start_us":112924,"threads_started":1,"txn-participant-registration.queue_time_us":101387,"txn-participant-registration.run_cpu_time_us":757,"txn-participant-registration.run_wall_time_us":110955,"child_traces":[["op",{"apply.queue_time_us":41,"apply.run_cpu_time_us":536,"apply.run_wall_time_us":699,"prepare.queue_time_us":1998428,"prepare.run_cpu_time_us":347,"prepare.run_wall_time_us":357,"replication_time_us":6037,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":3003132,"thread_start_us":243,"threads_started":1,"txn-participant-registration.queue_time_us":406}]]}]]}
W20251028 14:51:42.390233 1656 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
W20251028 14:51:42.390519 1567 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:42.390816 1603 txn_write_ops-itest.cc:454] Txn 2 wrote row 15: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:42.390899 1603 txn_write_ops-itest.cc:465] Txn 2 wrote row 15: Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:42.403858 1748 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:40817 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=5, attempt_no=0}) took 3000 ms. Trace:
I20251028 14:51:42.404021 1748 rpcz_store.cc:276] 1028 14:51:39.402962 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:39.403014 (+ 52us) service_pool.cc:225] Handling call
1028 14:51:42.403790 (+3000776us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:42.390148 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:42.390337 (+ 189us) participant_op.cc:199] PREPARE: finished
1028 14:51:42.390379 (+ 42us) participant_op.cc:212] START. Timestamp: 1761663102390340
1028 14:51:42.390381 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:42.390553 (+ 172us) log.cc:844] Serialized 65 byte log entry
1028 14:51:42.403281 (+ 12728us) op_driver.cc:464] REPLICATION: finished
1028 14:51:42.403450 (+ 169us) participant_op.cc:265] APPLY: starting
1028 14:51:42.403638 (+ 188us) participant_op.cc:270] APPLY: finished
1028 14:51:42.403702 (+ 64us) log.cc:844] Serialized 14 byte log entry
1028 14:51:42.403750 (+ 48us) participant_op.cc:94] Released txn lock
1028 14:51:42.403766 (+ 16us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":78,"prepare.queue_time_us":2986800,"prepare.run_cpu_time_us":10200,"prepare.run_wall_time_us":11603,"raft.queue_time_us":809,"raft.run_cpu_time_us":571,"raft.run_wall_time_us":571,"replication_time_us":12848,"spinlock_wait_cycles":1408,"thread_start_us":11021,"threads_started":2,"wal-append.queue_time_us":699}]]}
W20251028 14:51:42.474699 1657 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20251028 14:51:42.474951 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:42.475265 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=8, attempt_no=2}) took 1006 ms. Trace:
I20251028 14:51:42.475343 1657 rpcz_store.cc:276] 1028 14:51:41.469172 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:41.469292 (+ 120us) service_pool.cc:225] Handling call
1028 14:51:42.475211 (+1005919us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:41.473784 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:41.473903 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:41.473906 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:41.473908 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:41.474069 (+ 161us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:42.474893 (+1000824us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:42.474909 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:42.474918 (+ 9us) write_op.cc:454] Released schema lock
1028 14:51:42.474929 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":3895}]]}
I20251028 14:51:42.482251 1738 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=10, attempt_no=0}) took 1003 ms. Trace:
I20251028 14:51:42.482528 1738 rpcz_store.cc:276] 1028 14:51:41.479178 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:41.479236 (+ 58us) service_pool.cc:225] Handling call
1028 14:51:42.482163 (+1002927us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:42.475932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:42.476048 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:42.476050 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:51:42.476051 (+ 1us) tablet.cc:660] Decoding operations
1028 14:51:42.476123 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:42.476132 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1028 14:51:42.476134 (+ 2us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:51:42.476200 (+ 66us) tablet.cc:699] Row locks acquired
1028 14:51:42.476202 (+ 2us) write_op.cc:260] PREPARE: finished
1028 14:51:42.476246 (+ 44us) write_op.cc:270] Start()
1028 14:51:42.476278 (+ 32us) write_op.cc:276] Timestamp: P: 1761663102476210 usec, L: 0
1028 14:51:42.476281 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:42.476480 (+ 199us) log.cc:844] Serialized 302 byte log entry
1028 14:51:42.481380 (+ 4900us) op_driver.cc:464] REPLICATION: finished
1028 14:51:42.481553 (+ 173us) write_op.cc:301] APPLY: starting
1028 14:51:42.481581 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
1028 14:51:42.481640 (+ 59us) tablet.cc:1368] finished BulkCheckPresence
1028 14:51:42.481691 (+ 51us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:51:42.481764 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:51:42.481801 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1028 14:51:42.481808 (+ 7us) write_op.cc:312] APPLY: finished
1028 14:51:42.481902 (+ 94us) log.cc:844] Serialized 24 byte log entry
1028 14:51:42.481999 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:42.482021 (+ 22us) write_op.cc:454] Released schema lock
1028 14:51:42.482028 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":109,"num_ops":1,"prepare.queue_time_us":996190,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":682,"raft.queue_time_us":156,"raft.run_cpu_time_us":1664,"raft.run_wall_time_us":1664,"replication_time_us":5059,"spinlock_wait_cycles":90368}]]}
I20251028 14:51:42.483664 1606 txn_write_ops-itest.cc:454] Txn 4 wrote row 14: OK
W20251028 14:51:43.402719 1656 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:51:43.402956 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:43.403247 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=0, attempt_no=2}) took 3015 ms. Trace:
I20251028 14:51:43.403326 1656 rpcz_store.cc:276] 1028 14:51:40.388219 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:40.388295 (+ 76us) service_pool.cc:225] Handling call
1028 14:51:43.403194 (+3014899us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:42.401835 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:42.401975 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:42.401978 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:42.401980 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:42.402059 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:43.402904 (+1000845us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:43.402919 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:43.402925 (+ 6us) write_op.cc:454] Released schema lock
1028 14:51:43.402935 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":2012998}]]}
W20251028 14:51:43.403964 1656 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251028 14:51:43.409132 1759 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:40817 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=7, attempt_no=0}) took 2014 ms. Trace:
I20251028 14:51:43.409399 1759 rpcz_store.cc:276] 1028 14:51:41.394518 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:41.394572 (+ 54us) service_pool.cc:225] Handling call
1028 14:51:43.409060 (+2014488us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:43.403871 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:43.404083 (+ 212us) participant_op.cc:199] PREPARE: finished
1028 14:51:43.404133 (+ 50us) participant_op.cc:212] START. Timestamp: 1761663103404046
1028 14:51:43.404135 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:43.404328 (+ 193us) log.cc:844] Serialized 65 byte log entry
1028 14:51:43.408112 (+ 3784us) op_driver.cc:464] REPLICATION: finished
1028 14:51:43.408701 (+ 589us) participant_op.cc:265] APPLY: starting
1028 14:51:43.408873 (+ 172us) participant_op.cc:270] APPLY: finished
1028 14:51:43.408926 (+ 53us) log.cc:844] Serialized 14 byte log entry
1028 14:51:43.408967 (+ 41us) participant_op.cc:94] Released txn lock
1028 14:51:43.408982 (+ 15us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":508,"prepare.queue_time_us":2009000,"prepare.run_cpu_time_us":966,"prepare.run_wall_time_us":962,"raft.queue_time_us":349,"raft.run_cpu_time_us":721,"raft.run_wall_time_us":722,"replication_time_us":3941,"spinlock_wait_cycles":48512,"thread_start_us":1319,"threads_started":3,"wal-append.queue_time_us":248}]]}
I20251028 14:51:43.419629 1759 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:40817 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=8, attempt_no=0}) took 1023 ms. Trace:
I20251028 14:51:43.420600 1759 rpcz_store.cc:276] 1028 14:51:42.396477 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:42.396683 (+ 206us) service_pool.cc:225] Handling call
1028 14:51:43.419545 (+1022862us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:43.404880 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:43.404939 (+ 59us) participant_op.cc:199] PREPARE: finished
1028 14:51:43.404972 (+ 33us) participant_op.cc:212] START. Timestamp: 1761663103404892
1028 14:51:43.404975 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:51:43.405133 (+ 158us) log.cc:844] Serialized 65 byte log entry
1028 14:51:43.411871 (+ 6738us) op_driver.cc:464] REPLICATION: finished
1028 14:51:43.419159 (+ 7288us) participant_op.cc:265] APPLY: starting
1028 14:51:43.419375 (+ 216us) participant_op.cc:270] APPLY: finished
1028 14:51:43.419453 (+ 78us) log.cc:844] Serialized 14 byte log entry
1028 14:51:43.419495 (+ 42us) participant_op.cc:94] Released txn lock
1028 14:51:43.419510 (+ 15us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7183,"prepare.queue_time_us":1007891,"prepare.run_cpu_time_us":308,"prepare.run_wall_time_us":308,"replication_time_us":6864,"spinlock_wait_cycles":35968}]]}
W20251028 14:51:43.484193 1657 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20251028 14:51:43.484411 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:43.484661 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=8, attempt_no=3}) took 1002 ms. Trace:
I20251028 14:51:43.484733 1657 rpcz_store.cc:276] 1028 14:51:42.481900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:42.481970 (+ 70us) service_pool.cc:225] Handling call
1028 14:51:43.484614 (+1002644us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:42.483361 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:42.483500 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:42.483503 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:42.483505 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:42.483589 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:43.484361 (+1000772us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:43.484374 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:43.484381 (+ 7us) write_op.cc:454] Released schema lock
1028 14:51:43.484389 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"spinlock_wait_cycles":694528,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":488}]]}
W20251028 14:51:44.410313 1656 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:51:44.410626 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:44.410897 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=0, attempt_no=3}) took 1002 ms. Trace:
I20251028 14:51:44.411053 1656 rpcz_store.cc:276] 1028 14:51:43.408857 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:43.408911 (+ 54us) service_pool.cc:225] Handling call
1028 14:51:44.410863 (+1001952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:43.409490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:43.409618 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:43.409621 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:43.409624 (+ 3us) tablet.cc:660] Decoding operations
1028 14:51:43.409708 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:44.410505 (+1000797us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:44.410523 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:44.410532 (+ 9us) write_op.cc:454] Released schema lock
1028 14:51:44.410544 (+ 12us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":42}]]}
W20251028 14:51:44.492673 1657 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20251028 14:51:44.492926 1657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60816: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:44.493199 1657 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60816 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=8, attempt_no=4}) took 1002 ms. Trace:
I20251028 14:51:44.493369 1657 rpcz_store.cc:276] 1028 14:51:43.491129 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:43.491181 (+ 52us) service_pool.cc:225] Handling call
1028 14:51:44.493149 (+1001968us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:43.491731 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 7a7a7e704e324db593496fb85e9f5634
1028 14:51:43.491835 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:43.491838 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:43.491840 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:43.491911 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:44.492857 (+1000946us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:44.492880 (+ 23us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:44.492891 (+ 11us) write_op.cc:454] Released schema lock
1028 14:51:44.492901 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":50}]]}
I20251028 14:51:44.498236 1779 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:43983 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=10, attempt_no=0}) took 1002 ms. Trace:
I20251028 14:51:44.498549 1779 rpcz_store.cc:276] 1028 14:51:43.495461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:43.495614 (+ 153us) service_pool.cc:225] Handling call
1028 14:51:44.498088 (+1002474us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:44.493915 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:44.493973 (+ 58us) participant_op.cc:199] PREPARE: finished
1028 14:51:44.493991 (+ 18us) participant_op.cc:212] START. Timestamp: 1761663104493946
1028 14:51:44.493993 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:44.494154 (+ 161us) log.cc:844] Serialized 75 byte log entry
1028 14:51:44.497590 (+ 3436us) op_driver.cc:464] REPLICATION: finished
1028 14:51:44.497710 (+ 120us) participant_op.cc:265] APPLY: starting
1028 14:51:44.497896 (+ 186us) participant_op.cc:270] APPLY: finished
1028 14:51:44.497982 (+ 86us) log.cc:844] Serialized 14 byte log entry
1028 14:51:44.498044 (+ 62us) participant_op.cc:94] Released txn lock
1028 14:51:44.498063 (+ 19us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"prepare.queue_time_us":997990,"prepare.run_cpu_time_us":594,"prepare.run_wall_time_us":594,"raft.queue_time_us":124,"raft.run_cpu_time_us":642,"raft.run_wall_time_us":641,"replication_time_us":3559,"thread_start_us":225,"threads_started":1,"wal-append.queue_time_us":348}]]}
I20251028 14:51:44.507298 1601 txn_write_ops-itest.cc:454] Txn 1 wrote row 4: OK
W20251028 14:51:45.412442 1656 lock_manager.cc:494] Txn 8 has not acquired the partition lock after 1000ms
W20251028 14:51:45.413116 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:45.413228 1790 ts_tablet_manager.cc:2121] Sending abort request for transaction 8
I20251028 14:51:45.413496 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=11, attempt_no=0}) took 3024 ms. Trace:
I20251028 14:51:45.413612 1656 rpcz_store.cc:276] 1028 14:51:42.388776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:42.388843 (+ 67us) service_pool.cc:225] Handling call
1028 14:51:45.413448 (+3024605us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:43.405221 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:43.405283 (+ 62us) participant_op.cc:199] PREPARE: finished
1028 14:51:43.405307 (+ 24us) participant_op.cc:212] START. Timestamp: 1761663103405232
1028 14:51:43.405309 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:43.405411 (+ 102us) log.cc:844] Serialized 65 byte log entry
1028 14:51:43.411942 (+ 6531us) op_driver.cc:464] REPLICATION: finished
1028 14:51:43.433882 (+ 21940us) participant_op.cc:265] APPLY: starting
1028 14:51:43.434428 (+ 546us) participant_op.cc:270] APPLY: finished
1028 14:51:43.434503 (+ 75us) log.cc:844] Serialized 14 byte log entry
1028 14:51:43.434567 (+ 64us) participant_op.cc:94] Released txn lock
1028 14:51:43.434576 (+ 9us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:44.411650 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:44.411808 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:44.411811 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:44.411814 (+ 3us) tablet.cc:660] Decoding operations
1028 14:51:44.411895 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:45.412645 (+1000750us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:51:45.412659 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:45.412666 (+ 7us) write_op.cc:454] Released schema lock
1028 14:51:45.412676 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":300,"threads_started":1,"txn-participant-registration.queue_time_us":415,"txn-participant-registration.run_cpu_time_us":803,"txn-participant-registration.run_wall_time_us":9464,"child_traces":[["op",{"apply.queue_time_us":21886,"apply.run_cpu_time_us":1116,"apply.run_wall_time_us":3144,"prepare.queue_time_us":1006197,"prepare.run_cpu_time_us":224,"prepare.run_wall_time_us":222,"replication_time_us":6616,"thread_start_us":26874,"threads_started":1,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":976795,"thread_start_us":304,"threads_started":1,"txn-participant-registration.queue_time_us":444}]]}]]}
W20251028 14:51:45.414888 1567 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:45.415689 1602 txn_write_ops-itest.cc:454] Txn 8 wrote row 3: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:45.415823 1602 txn_write_ops-itest.cc:465] Txn 8 wrote row 3: Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:51:46.417129 1656 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:51:46.417495 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:51:46.417775 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=0, attempt_no=4}) took 2001 ms. Trace:
I20251028 14:51:46.417873 1656 rpcz_store.cc:276] 1028 14:51:44.416879 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:44.417063 (+ 184us) service_pool.cc:225] Handling call
1028 14:51:46.417738 (+2000675us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:45.416235 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:45.416365 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:45.416369 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:51:45.416371 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:45.416460 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:46.417432 (+1000972us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:51:46.417445 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:46.417452 (+ 7us) write_op.cc:454] Released schema lock
1028 14:51:46.417461 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":998501}]]}
W20251028 14:51:46.419900 1656 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations [suppressed 1 similar messages]
I20251028 14:51:46.423014 1811 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:40817 (ReqId={client: d1de460300ee460a8516117ebaf97cb2, seq_no=12, attempt_no=0}) took 1908 ms. Trace:
I20251028 14:51:46.423314 1811 rpcz_store.cc:276] 1028 14:51:44.514838 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:44.515032 (+ 194us) service_pool.cc:225] Handling call
1028 14:51:46.422925 (+1907893us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:46.418747 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:46.418815 (+ 68us) participant_op.cc:199] PREPARE: finished
1028 14:51:46.418893 (+ 78us) participant_op.cc:212] START. Timestamp: 1761663106418800
1028 14:51:46.418895 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:46.419112 (+ 217us) log.cc:844] Serialized 65 byte log entry
1028 14:51:46.422418 (+ 3306us) op_driver.cc:464] REPLICATION: finished
1028 14:51:46.422527 (+ 109us) participant_op.cc:265] APPLY: starting
1028 14:51:46.422611 (+ 84us) participant_op.cc:270] APPLY: finished
1028 14:51:46.422668 (+ 57us) log.cc:844] Serialized 14 byte log entry
1028 14:51:46.422891 (+ 223us) participant_op.cc:94] Released txn lock
1028 14:51:46.422902 (+ 11us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"mutex_wait_us":266,"prepare.queue_time_us":1903394,"prepare.run_cpu_time_us":1015,"prepare.run_wall_time_us":1016,"raft.queue_time_us":674,"raft.run_cpu_time_us":1260,"raft.run_wall_time_us":1259,"replication_time_us":3486,"thread_start_us":430,"threads_started":1}]]}
W20251028 14:51:46.426337 1567 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 6 attempt(s): Failed to write to server: (no server available): Write(tablet: 793dce1569ed41e3aad8035b9da60839, num_ops: 1, num_attempts: 6) passed its deadline
I20251028 14:51:46.427351 1599 txn_write_ops-itest.cc:454] Txn 0 wrote row 8: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:46.427449 1599 txn_write_ops-itest.cc:465] Txn 0 wrote row 8: Timed out: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 6 attempt(s): Failed to write to server: (no server available): Write(tablet: 793dce1569ed41e3aad8035b9da60839, num_ops: 1, num_attempts: 6) passed its deadline
W20251028 14:51:47.421609 1656 lock_manager.cc:494] Txn 9 has not acquired the partition lock after 1000ms
I20251028 14:51:47.422134 1830 ts_tablet_manager.cc:2121] Sending abort request for transaction 9
W20251028 14:51:47.422030 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:47.422454 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=12, attempt_no=0}) took 3026 ms. Trace:
I20251028 14:51:47.422559 1656 rpcz_store.cc:276] 1028 14:51:44.395565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:44.395699 (+ 134us) service_pool.cc:225] Handling call
1028 14:51:47.422381 (+3026682us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:45.414414 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:45.414455 (+ 41us) participant_op.cc:199] PREPARE: finished
1028 14:51:45.414494 (+ 39us) participant_op.cc:212] START. Timestamp: 1761663105414440
1028 14:51:45.414496 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:45.414668 (+ 172us) log.cc:844] Serialized 65 byte log entry
1028 14:51:45.421708 (+ 7040us) op_driver.cc:464] REPLICATION: finished
1028 14:51:45.426010 (+ 4302us) participant_op.cc:265] APPLY: starting
1028 14:51:45.427227 (+ 1217us) participant_op.cc:270] APPLY: finished
1028 14:51:45.427274 (+ 47us) log.cc:844] Serialized 14 byte log entry
1028 14:51:45.427491 (+ 217us) participant_op.cc:94] Released txn lock
1028 14:51:45.427500 (+ 9us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:46.420606 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:46.420789 (+ 183us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:46.420792 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:46.420795 (+ 3us) tablet.cc:660] Decoding operations
1028 14:51:46.420897 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:47.421811 (+1000914us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:51:47.421829 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:47.421836 (+ 7us) write_op.cc:454] Released schema lock
1028 14:51:47.421847 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":428,"threads_started":1,"txn-participant-registration.queue_time_us":506,"txn-participant-registration.run_cpu_time_us":834,"txn-participant-registration.run_wall_time_us":8242,"child_traces":[["op",{"apply.queue_time_us":4221,"apply.run_cpu_time_us":973,"apply.run_wall_time_us":1920,"prepare.queue_time_us":1009547,"prepare.run_cpu_time_us":835,"prepare.run_wall_time_us":1768,"raft.queue_time_us":813,"raft.run_cpu_time_us":616,"raft.run_wall_time_us":615,"replication_time_us":7181,"thread_start_us":5412,"threads_started":3,"wal-append.queue_time_us":340,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":992571,"txn-participant-registration.queue_time_us":96}]]}]]}
I20251028 14:51:47.423835 1604 txn_write_ops-itest.cc:454] Txn 9 wrote row 13: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:47.423938 1604 txn_write_ops-itest.cc:465] Txn 9 wrote row 13: Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:51:47.435169 1656 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251028 14:51:47.438153 1835 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=13, attempt_no=0}) took 1039 ms. Trace:
I20251028 14:51:47.438365 1835 rpcz_store.cc:276] 1028 14:51:46.398864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:46.398931 (+ 67us) service_pool.cc:225] Handling call
1028 14:51:47.438090 (+1039159us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:47.423310 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:47.423353 (+ 43us) participant_op.cc:199] PREPARE: finished
1028 14:51:47.423392 (+ 39us) participant_op.cc:212] START. Timestamp: 1761663107423368
1028 14:51:47.423394 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:47.423547 (+ 153us) log.cc:844] Serialized 65 byte log entry
1028 14:51:47.427386 (+ 3839us) op_driver.cc:464] REPLICATION: finished
1028 14:51:47.427467 (+ 81us) participant_op.cc:265] APPLY: starting
1028 14:51:47.427771 (+ 304us) participant_op.cc:270] APPLY: finished
1028 14:51:47.427806 (+ 35us) log.cc:844] Serialized 14 byte log entry
1028 14:51:47.427840 (+ 34us) participant_op.cc:94] Released txn lock
1028 14:51:47.427847 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:47.428203 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:47.428297 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:47.428300 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:47.428302 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:47.428372 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:47.432410 (+ 4038us) write_op.cc:641] Partition lock acquired for write op
1028 14:51:47.432417 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:51:47.432485 (+ 68us) tablet.cc:699] Row locks acquired
1028 14:51:47.432487 (+ 2us) write_op.cc:260] PREPARE: finished
1028 14:51:47.432529 (+ 42us) write_op.cc:270] Start()
1028 14:51:47.432557 (+ 28us) write_op.cc:276] Timestamp: P: 1761663107432512 usec, L: 0
1028 14:51:47.432561 (+ 4us) op_driver.cc:348] REPLICATION: starting
1028 14:51:47.434675 (+ 2114us) log.cc:844] Serialized 302 byte log entry
1028 14:51:47.434974 (+ 299us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x6160000a0954. stack: 00007f1f2600b9f6 00007f1f2600b7b0 00007f1f24fe73ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1f31eab862 00007f1f3328e2a5 00007f1f3328cdc9 00007f1f332951f0 00007f1f33294ffd 0000000000a8867c 00007f1f26051d7f 00007f1f2605504b 00007f1f26054ec8 0000000000a8867c
1028 14:51:47.437299 (+ 2325us) op_driver.cc:464] REPLICATION: finished
1028 14:51:47.437436 (+ 137us) write_op.cc:301] APPLY: starting
1028 14:51:47.437466 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
1028 14:51:47.437526 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
1028 14:51:47.437528 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:51:47.437682 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:51:47.437719 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1028 14:51:47.437725 (+ 6us) write_op.cc:312] APPLY: finished
1028 14:51:47.437829 (+ 104us) log.cc:844] Serialized 24 byte log entry
1028 14:51:47.437930 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:47.437946 (+ 16us) write_op.cc:454] Released schema lock
1028 14:51:47.437953 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"thread_start_us":248,"threads_started":1,"txn-participant-registration.queue_time_us":351,"txn-participant-registration.run_cpu_time_us":737,"txn-participant-registration.run_wall_time_us":6475,"child_traces":[["op",{"apply.queue_time_us":34,"apply.run_cpu_time_us":687,"apply.run_wall_time_us":686,"prepare.queue_time_us":1017266,"prepare.run_cpu_time_us":507,"prepare.run_wall_time_us":1081,"raft.queue_time_us":702,"raft.run_cpu_time_us":589,"raft.run_wall_time_us":588,"replication_time_us":3962,"spinlock_wait_cycles":21376,"thread_start_us":706,"threads_started":1,"child_traces":[["op",{"apply.queue_time_us":66,"mutex_wait_us":887,"num_ops":1,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":1015,"prepare.run_wall_time_us":6840,"raft.queue_time_us":207,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":1561,"replication_time_us":4708,"spinlock_wait_cycles":4350592}]]}]]}
I20251028 14:51:47.441464 1605 txn_write_ops-itest.cc:454] Txn 10 wrote row 16: OK
I20251028 14:51:47.463034 1605 txn_write_ops-itest.cc:454] Txn 10 wrote row 6: OK
W20251028 14:51:48.436424 1656 lock_manager.cc:494] Txn 11 has not acquired the partition lock after 1000ms
W20251028 14:51:48.437098 1656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:57884: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:48.437306 1842 ts_tablet_manager.cc:2121] Sending abort request for transaction 11
I20251028 14:51:48.437477 1656 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:57884 (ReqId={client: b51552c17bb24f98a98dcd7fb9c706b6, seq_no=14, attempt_no=0}) took 1036 ms. Trace:
I20251028 14:51:48.437590 1656 rpcz_store.cc:276] 1028 14:51:47.400893 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:51:47.400970 (+ 77us) service_pool.cc:225] Handling call
1028 14:51:48.437431 (+1036461us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:51:47.424784 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:51:47.424815 (+ 31us) participant_op.cc:199] PREPARE: finished
1028 14:51:47.424838 (+ 23us) participant_op.cc:212] START. Timestamp: 1761663107424819
1028 14:51:47.424840 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:51:47.424940 (+ 100us) log.cc:844] Serialized 65 byte log entry
1028 14:51:47.431261 (+ 6321us) op_driver.cc:464] REPLICATION: finished
1028 14:51:47.431810 (+ 549us) participant_op.cc:265] APPLY: starting
1028 14:51:47.432043 (+ 233us) participant_op.cc:270] APPLY: finished
1028 14:51:47.432082 (+ 39us) log.cc:844] Serialized 14 byte log entry
1028 14:51:47.432111 (+ 29us) participant_op.cc:94] Released txn lock
1028 14:51:47.432117 (+ 6us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:51:47.435573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 793dce1569ed41e3aad8035b9da60839
1028 14:51:47.435692 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:51:47.435695 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:51:47.435697 (+ 2us) tablet.cc:660] Decoding operations
1028 14:51:47.435781 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:51:48.436610 (+1000829us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:51:48.436623 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:51:48.436630 (+ 7us) write_op.cc:454] Released schema lock
1028 14:51:48.436640 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":226,"threads_started":1,"txn-participant-registration.queue_time_us":313,"txn-participant-registration.run_cpu_time_us":644,"txn-participant-registration.run_wall_time_us":6328,"child_traces":[["op",{"apply.queue_time_us":518,"apply.run_cpu_time_us":565,"apply.run_wall_time_us":565,"prepare.queue_time_us":16808,"prepare.run_cpu_time_us":190,"prepare.run_wall_time_us":188,"replication_time_us":6404,"thread_start_us":3021,"threads_started":1,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":3229,"thread_start_us":292,"threads_started":1,"txn-participant-registration.queue_time_us":488}]]}]]}
W20251028 14:51:48.438656 1567 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:51:48.439407 1603 txn_write_ops-itest.cc:454] Txn 11 wrote row 15: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20251028 14:51:48.439517 1603 txn_write_ops-itest.cc:465] Txn 11 wrote row 15: Aborted: Failed to write batch of 1 ops to tablet 793dce1569ed41e3aad8035b9da60839 after 1 attempt(s): Failed to write to server: 07f8e2e889374d5ab3fd4938b9ced456 (127.0.104.1:40787): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:51:48.453591 1656 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251028 14:51:50.446491 1602 txn_write_ops-itest.cc:454] Txn 12 wrote row 3: OK
I20251028 14:51:50.452858 1602 txn_write_ops-itest.cc:454] Txn 12 wrote row 2: OK
I20251028 14:51:51.459085 1599 txn_write_ops-itest.cc:454] Txn 13 wrote row 8: OK
I20251028 14:51:51.480561 1599 txn_write_ops-itest.cc:454] Txn 13 wrote row 9: OK
I20251028 14:51:52.454154 1604 txn_write_ops-itest.cc:454] Txn 14 wrote row 13: OK
I20251028 14:51:52.470026 1604 txn_write_ops-itest.cc:454] Txn 14 wrote row 5: OK
I20251028 14:51:53.471567 1603 txn_write_ops-itest.cc:454] Txn 15 wrote row 15: OK
I20251028 14:51:53.488287 1603 txn_write_ops-itest.cc:454] Txn 15 wrote row 1: OK
I20251028 14:51:53.564718 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 1149
I20251028 14:51:53.568127 1279 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:53.686883 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 1149
I20251028 14:51:53.705333 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 1286
I20251028 14:51:53.706385 1421 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:53.829759 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 1286
I20251028 14:51:53.846589 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 1428
I20251028 14:51:53.848248 1557 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:54.011402 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 1428
I20251028 14:51:54.033082 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 1044
I20251028 14:51:54.034340 1115 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:54.155582 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 1044
2025-10-28T14:51:54Z chronyd exiting
[ OK ] TxnWriteOpsITest.DeadlockPrevention (21187 ms)
[ RUN ] TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds
2025-10-28T14:51:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:51:54Z Disabled control of system clock
I20251028 14:51:54.227804 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33425
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33425
--txn_manager_enabled=true
--txn_manager_status_table_range_partition_span=3 with env {}
W20251028 14:51:54.585345 1941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:54.585685 1941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:54.585765 1941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:54.594172 1941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:51:54.594347 1941 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:51:54.594396 1941 flags.cc:425] Enabled experimental flag: --txn_manager_status_table_range_partition_span=3
W20251028 14:51:54.594431 1941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:54.594491 1941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:51:54.594539 1941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:51:54.604758 1941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33425
--txn_manager_enabled=true
--txn_manager_status_table_range_partition_span=3
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:54.606587 1941 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:54.608779 1941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:54.618469 1947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:54.618639 1946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:54.618598 1949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:54.620133 1941 server_base.cc:1047] running on GCE node
I20251028 14:51:54.621219 1941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:54.622895 1941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:54.624153 1941 hybrid_clock.cc:648] HybridClock initialized: now 1761663114624083 us; error 60 us; skew 500 ppm
I20251028 14:51:54.624596 1941 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:54.627151 1941 webserver.cc:492] Webserver started at http://127.0.104.62:37815/ using document root <none> and password file <none>
I20251028 14:51:54.627924 1941 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:54.628019 1941 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:54.628342 1941 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:54.630930 1941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "5c6e50781c034dd7ac5eb474140d1b0c"
format_stamp: "Formatted at 2025-10-28 14:51:54 on dist-test-slave-0v9m"
I20251028 14:51:54.631783 1941 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "5c6e50781c034dd7ac5eb474140d1b0c"
format_stamp: "Formatted at 2025-10-28 14:51:54 on dist-test-slave-0v9m"
I20251028 14:51:54.638540 1941 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20251028 14:51:54.644445 1955 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:54.646545 1941 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20251028 14:51:54.646775 1941 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "5c6e50781c034dd7ac5eb474140d1b0c"
format_stamp: "Formatted at 2025-10-28 14:51:54 on dist-test-slave-0v9m"
I20251028 14:51:54.647028 1941 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:54.671648 1941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:54.672617 1941 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:54.672892 1941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:54.697718 1941 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33425
I20251028 14:51:54.697737 2016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33425 every 8 connection(s)
I20251028 14:51:54.699273 1941 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:51:54.700847 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 1941
I20251028 14:51:54.701095 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:51:54.705557 2017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:54.716413 2017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: Bootstrap starting.
I20251028 14:51:54.719859 2017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:54.720952 2017 log.cc:826] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:54.724385 2017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: No bootstrap required, opened a new log
I20251028 14:51:54.729892 2017 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33425 } }
I20251028 14:51:54.730286 2017 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:54.730394 2017 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6e50781c034dd7ac5eb474140d1b0c, State: Initialized, Role: FOLLOWER
I20251028 14:51:54.731029 2017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33425 } }
I20251028 14:51:54.731241 2017 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:51:54.731379 2017 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:51:54.731537 2017 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:54.733762 2017 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33425 } }
I20251028 14:51:54.734321 2017 leader_election.cc:304] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c6e50781c034dd7ac5eb474140d1b0c; no voters:
I20251028 14:51:54.734972 2017 leader_election.cc:290] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:51:54.735129 2022 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:54.735795 2022 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [term 1 LEADER]: Becoming Leader. State: Replica: 5c6e50781c034dd7ac5eb474140d1b0c, State: Running, Role: LEADER
I20251028 14:51:54.736366 2022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33425 } }
I20251028 14:51:54.737035 2017 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:51:54.739666 2024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c6e50781c034dd7ac5eb474140d1b0c. Latest consensus state: current_term: 1 leader_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33425 } } }
I20251028 14:51:54.739655 2023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6e50781c034dd7ac5eb474140d1b0c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33425 } } }
I20251028 14:51:54.740049 2023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [sys.catalog]: This master's current role is: LEADER
I20251028 14:51:54.740047 2024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c [sys.catalog]: This master's current role is: LEADER
I20251028 14:51:54.741864 2030 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:51:54.746831 2030 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:51:54.757369 2030 catalog_manager.cc:1357] Generated new cluster ID: 9d73fd04a9a54e44b0ba50bb500bea83
I20251028 14:51:54.757603 2030 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:51:54.779822 2030 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:51:54.781764 2030 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:51:54.793665 2030 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: Generated new TSK 0
I20251028 14:51:54.794958 2030 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:51:54.803133 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33425
--builtin_ntp_servers=127.0.104.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:51:55.159168 2041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:55.159523 2041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:55.159639 2041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:55.168207 2041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:55.168383 2041 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:55.168488 2041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:51:55.179406 2041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:33425
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:55.181139 2041 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:55.183796 2041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:55.196781 2046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:55.196761 2047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:55.197180 2049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:55.198364 2041 server_base.cc:1047] running on GCE node
I20251028 14:51:55.199040 2041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:55.200112 2041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:55.201542 2041 hybrid_clock.cc:648] HybridClock initialized: now 1761663115201470 us; error 46 us; skew 500 ppm
I20251028 14:51:55.201870 2041 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:55.204450 2041 webserver.cc:492] Webserver started at http://127.0.104.1:32859/ using document root <none> and password file <none>
I20251028 14:51:55.205325 2041 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:55.205459 2041 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:55.205816 2041 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:55.208405 2041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
format_stamp: "Formatted at 2025-10-28 14:51:55 on dist-test-slave-0v9m"
I20251028 14:51:55.209233 2041 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
format_stamp: "Formatted at 2025-10-28 14:51:55 on dist-test-slave-0v9m"
I20251028 14:51:55.215329 2041 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251028 14:51:55.219717 2055 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:55.221510 2041 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:51:55.221690 2041 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
format_stamp: "Formatted at 2025-10-28 14:51:55 on dist-test-slave-0v9m"
I20251028 14:51:55.221880 2041 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:55.236572 2041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:55.237732 2041 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:55.238063 2041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:55.241956 2041 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:55.242103 2041 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:55.242238 2041 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:55.242321 2041 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:55.288687 2041 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37033
I20251028 14:51:55.288738 2175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37033 every 8 connection(s)
I20251028 14:51:55.290197 2041 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:51:55.292702 2176 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33425
I20251028 14:51:55.293015 2176 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:55.294003 2176 heartbeater.cc:507] Master 127.0.104.62:33425 requested a full tablet report, sending...
I20251028 14:51:55.295408 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 2041
I20251028 14:51:55.295639 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:51:55.296231 1972 ts_manager.cc:194] Registered new tserver with Master: 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033)
I20251028 14:51:55.298537 1972 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:58187
I20251028 14:51:55.300230 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33425
--builtin_ntp_servers=127.0.104.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:51:55.655514 2178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:55.655874 2178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:55.656002 2178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:55.664793 2178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:55.664989 2178 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:55.665107 2178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:51:55.675387 2178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:33425
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:55.677109 2178 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:55.679205 2178 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:55.689916 2184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:55.689975 2183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:55.689953 2186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:55.690944 2178 server_base.cc:1047] running on GCE node
I20251028 14:51:55.691664 2178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:55.692719 2178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:55.693984 2178 hybrid_clock.cc:648] HybridClock initialized: now 1761663115693917 us; error 47 us; skew 500 ppm
I20251028 14:51:55.694329 2178 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:55.697098 2178 webserver.cc:492] Webserver started at http://127.0.104.2:37773/ using document root <none> and password file <none>
I20251028 14:51:55.698006 2178 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:55.698139 2178 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:55.698488 2178 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:55.701164 2178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "e55bd2b7d0a74b9bbaef40478491d5ca"
format_stamp: "Formatted at 2025-10-28 14:51:55 on dist-test-slave-0v9m"
I20251028 14:51:55.702068 2178 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "e55bd2b7d0a74b9bbaef40478491d5ca"
format_stamp: "Formatted at 2025-10-28 14:51:55 on dist-test-slave-0v9m"
I20251028 14:51:55.708073 2178 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251028 14:51:55.711922 2192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:55.713500 2178 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:51:55.713663 2178 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "e55bd2b7d0a74b9bbaef40478491d5ca"
format_stamp: "Formatted at 2025-10-28 14:51:55 on dist-test-slave-0v9m"
I20251028 14:51:55.713845 2178 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:55.743630 2178 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:55.744585 2178 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:55.744874 2178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:55.748242 2178 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:55.748370 2178 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:55.748485 2178 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:55.748565 2178 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:55.793808 2178 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:40751
I20251028 14:51:55.793888 2311 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:40751 every 8 connection(s)
I20251028 14:51:55.795210 2178 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:51:55.797569 2312 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33425
I20251028 14:51:55.797875 2312 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:55.798708 2312 heartbeater.cc:507] Master 127.0.104.62:33425 requested a full tablet report, sending...
I20251028 14:51:55.800423 1972 ts_manager.cc:194] Registered new tserver with Master: e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2:40751)
I20251028 14:51:55.801120 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 2178
I20251028 14:51:55.801375 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:51:55.801525 1972 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:38219
I20251028 14:51:55.804867 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33425
--builtin_ntp_servers=127.0.104.20:35547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:51:56.161542 2314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:56.162125 2314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:56.162539 2314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:56.171650 2314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:56.171833 2314 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:51:56.171957 2314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:51:56.182493 2314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:33425
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:56.184228 2314 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:56.186307 2314 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:56.196445 2320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:56.196511 2319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:56.197144 2322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:56.197880 2314 server_base.cc:1047] running on GCE node
I20251028 14:51:56.198565 2314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:56.199549 2314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:56.200814 2314 hybrid_clock.cc:648] HybridClock initialized: now 1761663116200745 us; error 39 us; skew 500 ppm
I20251028 14:51:56.201139 2314 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:56.203783 2314 webserver.cc:492] Webserver started at http://127.0.104.3:36459/ using document root <none> and password file <none>
I20251028 14:51:56.204582 2314 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:56.204703 2314 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:56.205067 2314 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:56.207672 2314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "05e78f2c00064213b0e0102302749a51"
format_stamp: "Formatted at 2025-10-28 14:51:56 on dist-test-slave-0v9m"
I20251028 14:51:56.208524 2314 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "05e78f2c00064213b0e0102302749a51"
format_stamp: "Formatted at 2025-10-28 14:51:56 on dist-test-slave-0v9m"
I20251028 14:51:56.214597 2314 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20251028 14:51:56.218448 2328 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:56.220067 2314 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251028 14:51:56.220263 2314 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "05e78f2c00064213b0e0102302749a51"
format_stamp: "Formatted at 2025-10-28 14:51:56 on dist-test-slave-0v9m"
I20251028 14:51:56.220463 2314 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:56.239743 2314 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:56.240648 2314 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:56.240942 2314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:51:56.244118 2314 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:51:56.244252 2314 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:56.244379 2314 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:51:56.244472 2314 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:56.294847 2314 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:39965
I20251028 14:51:56.294940 2447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:39965 every 8 connection(s)
I20251028 14:51:56.296798 2314 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:51:56.299889 2448 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33425
I20251028 14:51:56.300252 2448 heartbeater.cc:461] Registering TS with master...
I20251028 14:51:56.301242 2448 heartbeater.cc:507] Master 127.0.104.62:33425 requested a full tablet report, sending...
I20251028 14:51:56.302700 2176 heartbeater.cc:499] Master 127.0.104.62:33425 was elected leader, sending a full tablet report...
I20251028 14:51:56.303481 1972 ts_manager.cc:194] Registered new tserver with Master: 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965)
I20251028 14:51:56.304533 1972 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:40203
I20251028 14:51:56.304677 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 2314
I20251028 14:51:56.304886 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:51:56.309893 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:51:56.331357 1972 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41702:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:51:56.336908 1972 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:51:56.370513 2241 tablet_service.cc:1505] Processing CreateTablet for tablet 64cf82de43d447a7ab38ff8ccf253d81 (DEFAULT_TABLE table=txn_write_ops_test [id=54a9a44f018842e685c1625e57040f40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:56.370447 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 311bcdeaa5d4466db7a06af60ba0cafd (DEFAULT_TABLE table=txn_write_ops_test [id=54a9a44f018842e685c1625e57040f40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:56.372588 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cf82de43d447a7ab38ff8ccf253d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.372156 2109 tablet_service.cc:1505] Processing CreateTablet for tablet 64cf82de43d447a7ab38ff8ccf253d81 (DEFAULT_TABLE table=txn_write_ops_test [id=54a9a44f018842e685c1625e57040f40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:56.373386 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 311bcdeaa5d4466db7a06af60ba0cafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.373732 2109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cf82de43d447a7ab38ff8ccf253d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.374068 2382 tablet_service.cc:1505] Processing CreateTablet for tablet 64cf82de43d447a7ab38ff8ccf253d81 (DEFAULT_TABLE table=txn_write_ops_test [id=54a9a44f018842e685c1625e57040f40]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:56.376214 2382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64cf82de43d447a7ab38ff8ccf253d81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.381649 2110 tablet_service.cc:1505] Processing CreateTablet for tablet 311bcdeaa5d4466db7a06af60ba0cafd (DEFAULT_TABLE table=txn_write_ops_test [id=54a9a44f018842e685c1625e57040f40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:56.382332 2110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 311bcdeaa5d4466db7a06af60ba0cafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.382436 2383 tablet_service.cc:1505] Processing CreateTablet for tablet 311bcdeaa5d4466db7a06af60ba0cafd (DEFAULT_TABLE table=txn_write_ops_test [id=54a9a44f018842e685c1625e57040f40]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:51:56.383080 2383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 311bcdeaa5d4466db7a06af60ba0cafd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.397576 2465 tablet_bootstrap.cc:492] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca: Bootstrap starting.
I20251028 14:51:56.399021 2466 tablet_bootstrap.cc:492] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1: Bootstrap starting.
I20251028 14:51:56.402077 2467 tablet_bootstrap.cc:492] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51: Bootstrap starting.
I20251028 14:51:56.402484 2465 tablet_bootstrap.cc:654] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.402478 2466 tablet_bootstrap.cc:654] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.403955 2465 log.cc:826] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:56.403975 2466 log.cc:826] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:56.405445 2467 tablet_bootstrap.cc:654] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.406710 2467 log.cc:826] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51: Log is configured to *not* fsync() on all Append() calls
I20251028 14:51:56.409597 2467 tablet_bootstrap.cc:492] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51: No bootstrap required, opened a new log
I20251028 14:51:56.409765 2466 tablet_bootstrap.cc:492] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1: No bootstrap required, opened a new log
I20251028 14:51:56.409909 2467 ts_tablet_manager.cc:1403] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20251028 14:51:56.410025 2466 ts_tablet_manager.cc:1403] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20251028 14:51:56.415320 2467 raft_consensus.cc:359] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.415786 2467 raft_consensus.cc:385] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.415608 2466 raft_consensus.cc:359] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.415896 2467 raft_consensus.cc:740] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.415982 2466 raft_consensus.cc:385] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.415867 2465 tablet_bootstrap.cc:492] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca: No bootstrap required, opened a new log
I20251028 14:51:56.416066 2466 raft_consensus.cc:740] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cf821b8d8e4a708d6e6de2516be3c1, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.416724 2467 consensus_queue.cc:260] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.416736 2466 consensus_queue.cc:260] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.417156 2465 ts_tablet_manager.cc:1403] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.006s
I20251028 14:51:56.418741 2448 heartbeater.cc:499] Master 127.0.104.62:33425 was elected leader, sending a full tablet report...
I20251028 14:51:56.418908 2466 ts_tablet_manager.cc:1434] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20251028 14:51:56.419363 2466 tablet_bootstrap.cc:492] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1: Bootstrap starting.
I20251028 14:51:56.421463 2467 ts_tablet_manager.cc:1434] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20251028 14:51:56.422029 2467 tablet_bootstrap.cc:492] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51: Bootstrap starting.
I20251028 14:51:56.422237 2466 tablet_bootstrap.cc:654] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.424113 2467 tablet_bootstrap.cc:654] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.426033 2465 raft_consensus.cc:359] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.426726 2465 raft_consensus.cc:385] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.426855 2465 raft_consensus.cc:740] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.427896 2465 consensus_queue.cc:260] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.431819 2312 heartbeater.cc:499] Master 127.0.104.62:33425 was elected leader, sending a full tablet report...
I20251028 14:51:56.432855 2465 ts_tablet_manager.cc:1434] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent starting tablet: real 0.015s user 0.010s sys 0.002s
I20251028 14:51:56.433403 2465 tablet_bootstrap.cc:492] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca: Bootstrap starting.
I20251028 14:51:56.434047 2466 tablet_bootstrap.cc:492] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1: No bootstrap required, opened a new log
I20251028 14:51:56.434237 2466 ts_tablet_manager.cc:1403] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20251028 14:51:56.435254 2465 tablet_bootstrap.cc:654] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.435314 2466 raft_consensus.cc:359] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.435536 2466 raft_consensus.cc:385] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.435623 2466 raft_consensus.cc:740] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cf821b8d8e4a708d6e6de2516be3c1, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.435930 2466 consensus_queue.cc:260] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.439237 2466 ts_tablet_manager.cc:1434] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251028 14:51:56.440037 2467 tablet_bootstrap.cc:492] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51: No bootstrap required, opened a new log
I20251028 14:51:56.440073 2465 tablet_bootstrap.cc:492] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca: No bootstrap required, opened a new log
I20251028 14:51:56.440207 2467 ts_tablet_manager.cc:1403] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
I20251028 14:51:56.440207 2465 ts_tablet_manager.cc:1403] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20251028 14:51:56.441030 2467 raft_consensus.cc:359] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.441030 2465 raft_consensus.cc:359] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.441239 2467 raft_consensus.cc:385] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.441239 2465 raft_consensus.cc:385] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.441380 2467 raft_consensus.cc:740] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.441380 2465 raft_consensus.cc:740] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.441623 2467 consensus_queue.cc:260] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.441623 2465 consensus_queue.cc:260] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.442138 2465 ts_tablet_manager.cc:1434] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:51:56.442138 2467 ts_tablet_manager.cc:1434] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251028 14:51:56.542739 2177 tablet.cc:2378] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:51:56.547274 2313 tablet.cc:2378] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:51:56.548650 2449 tablet.cc:2378] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:51:56.573935 2471 raft_consensus.cc:493] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:56.574223 2471 raft_consensus.cc:515] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.576921 2471 leader_election.cc:290] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2:40751), 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965)
I20251028 14:51:56.584569 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64cf82de43d447a7ab38ff8ccf253d81" candidate_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51" is_pre_election: true
I20251028 14:51:56.584579 2267 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64cf82de43d447a7ab38ff8ccf253d81" candidate_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" is_pre_election: true
I20251028 14:51:56.585084 2267 raft_consensus.cc:2468] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73cf821b8d8e4a708d6e6de2516be3c1 in term 0.
I20251028 14:51:56.585083 2403 raft_consensus.cc:2468] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73cf821b8d8e4a708d6e6de2516be3c1 in term 0.
I20251028 14:51:56.585819 2056 leader_election.cc:304] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:56.586246 2471 raft_consensus.cc:2804] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:56.586381 2471 raft_consensus.cc:493] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:56.586501 2471 raft_consensus.cc:3060] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.588701 2471 raft_consensus.cc:515] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.589428 2471 leader_election.cc:290] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [CANDIDATE]: Term 1 election: Requested vote from peers e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2:40751), 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965)
I20251028 14:51:56.589951 2267 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64cf82de43d447a7ab38ff8ccf253d81" candidate_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca"
I20251028 14:51:56.589963 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64cf82de43d447a7ab38ff8ccf253d81" candidate_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51"
I20251028 14:51:56.590135 2403 raft_consensus.cc:3060] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.590142 2267 raft_consensus.cc:3060] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.592439 2267 raft_consensus.cc:2468] T 64cf82de43d447a7ab38ff8ccf253d81 P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73cf821b8d8e4a708d6e6de2516be3c1 in term 1.
I20251028 14:51:56.592453 2403 raft_consensus.cc:2468] T 64cf82de43d447a7ab38ff8ccf253d81 P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73cf821b8d8e4a708d6e6de2516be3c1 in term 1.
I20251028 14:51:56.592978 2056 leader_election.cc:304] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:56.593361 2471 raft_consensus.cc:2804] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:56.593858 2471 raft_consensus.cc:697] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 LEADER]: Becoming Leader. State: Replica: 73cf821b8d8e4a708d6e6de2516be3c1, State: Running, Role: LEADER
I20251028 14:51:56.594388 2471 consensus_queue.cc:237] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.599578 1972 catalog_manager.cc:5649] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } health_report { overall_health: UNKNOWN } } }
I20251028 14:51:56.645285 2475 raft_consensus.cc:493] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:56.645514 2475 raft_consensus.cc:515] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.647238 2475 leader_election.cc:290] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965), 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033)
I20251028 14:51:56.654852 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "311bcdeaa5d4466db7a06af60ba0cafd" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51" is_pre_election: true
I20251028 14:51:56.655100 2403 raft_consensus.cc:2468] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 0.
I20251028 14:51:56.655619 2194 leader_election.cc:304] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:56.656028 2475 raft_consensus.cc:2804] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:56.656153 2475 raft_consensus.cc:493] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:56.655997 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "311bcdeaa5d4466db7a06af60ba0cafd" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" is_pre_election: true
I20251028 14:51:56.656243 2475 raft_consensus.cc:3060] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.656455 2131 raft_consensus.cc:2468] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 0.
I20251028 14:51:56.658571 2475 raft_consensus.cc:515] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.659142 2475 leader_election.cc:290] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 election: Requested vote from peers 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965), 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033)
I20251028 14:51:56.659751 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "311bcdeaa5d4466db7a06af60ba0cafd" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51"
I20251028 14:51:56.659842 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "311bcdeaa5d4466db7a06af60ba0cafd" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
I20251028 14:51:56.659974 2403 raft_consensus.cc:3060] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.660041 2131 raft_consensus.cc:3060] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.662045 2403 raft_consensus.cc:2468] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 1.
I20251028 14:51:56.662218 2131 raft_consensus.cc:2468] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 1.
I20251028 14:51:56.662590 2194 leader_election.cc:304] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:56.662964 2475 raft_consensus.cc:2804] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:56.663468 2475 raft_consensus.cc:697] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 LEADER]: Becoming Leader. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Running, Role: LEADER
I20251028 14:51:56.664000 2475 consensus_queue.cc:237] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } }
I20251028 14:51:56.668810 1971 catalog_manager.cc:5649] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } health_report { overall_health: UNKNOWN } } }
I20251028 14:51:56.798851 1971 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41718:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001\003\000\000\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:51:56.800263 1971 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:51:56.808615 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 74b5785bff29430c85a5bc0fc6eb604c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 3
I20251028 14:51:56.808502 2110 tablet_service.cc:1505] Processing CreateTablet for tablet 74b5785bff29430c85a5bc0fc6eb604c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 3
I20251028 14:51:56.809582 2110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b5785bff29430c85a5bc0fc6eb604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.809587 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b5785bff29430c85a5bc0fc6eb604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.812763 2383 tablet_service.cc:1505] Processing CreateTablet for tablet 74b5785bff29430c85a5bc0fc6eb604c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 3
I20251028 14:51:56.813728 2383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74b5785bff29430c85a5bc0fc6eb604c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:56.818050 2466 tablet_bootstrap.cc:492] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: Bootstrap starting.
I20251028 14:51:56.818050 2465 tablet_bootstrap.cc:492] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: Bootstrap starting.
I20251028 14:51:56.820631 2467 tablet_bootstrap.cc:492] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: Bootstrap starting.
I20251028 14:51:56.820639 2465 tablet_bootstrap.cc:654] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.820639 2466 tablet_bootstrap.cc:654] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.822504 2467 tablet_bootstrap.cc:654] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:56.824224 2466 tablet_bootstrap.cc:492] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: No bootstrap required, opened a new log
I20251028 14:51:56.824465 2466 ts_tablet_manager.cc:1403] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20251028 14:51:56.824924 2465 tablet_bootstrap.cc:492] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: No bootstrap required, opened a new log
I20251028 14:51:56.825088 2467 tablet_bootstrap.cc:492] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: No bootstrap required, opened a new log
I20251028 14:51:56.825130 2465 ts_tablet_manager.cc:1403] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20251028 14:51:56.825412 2466 raft_consensus.cc:359] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.825659 2466 raft_consensus.cc:385] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.825740 2466 raft_consensus.cc:740] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cf821b8d8e4a708d6e6de2516be3c1, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.826007 2466 consensus_queue.cc:260] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.825534 2467 ts_tablet_manager.cc:1403] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251028 14:51:56.826097 2465 raft_consensus.cc:359] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.826300 2465 raft_consensus.cc:385] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.826370 2465 raft_consensus.cc:740] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.826535 2466 ts_tablet_manager.cc:1434] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:51:56.826685 2465 consensus_queue.cc:260] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.827242 2465 ts_tablet_manager.cc:1434] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:51:56.827270 2467 raft_consensus.cc:359] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.827498 2467 raft_consensus.cc:385] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:56.827575 2467 raft_consensus.cc:740] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Initialized, Role: FOLLOWER
I20251028 14:51:56.827761 2475 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:56.827888 2467 consensus_queue.cc:260] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.828022 2475 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:56.828374 2467 ts_tablet_manager.cc:1434] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 14:51:56.828624 2472 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:56.828848 2472 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:56.829536 2471 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:56.829799 2471 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:56.959043 2475 raft_consensus.cc:493] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:56.959267 2475 raft_consensus.cc:515] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.959944 2475 leader_election.cc:290] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965)
I20251028 14:51:56.960566 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74b5785bff29430c85a5bc0fc6eb604c" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" is_pre_election: true
I20251028 14:51:56.960690 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74b5785bff29430c85a5bc0fc6eb604c" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51" is_pre_election: true
I20251028 14:51:56.960796 2131 raft_consensus.cc:2468] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 0.
I20251028 14:51:56.960922 2403 raft_consensus.cc:2468] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 0.
I20251028 14:51:56.961289 2194 leader_election.cc:304] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:56.961603 2475 raft_consensus.cc:2804] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:56.961694 2475 raft_consensus.cc:493] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:56.961810 2475 raft_consensus.cc:3060] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.963764 2475 raft_consensus.cc:515] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.964355 2475 leader_election.cc:290] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 election: Requested vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965)
I20251028 14:51:56.964913 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74b5785bff29430c85a5bc0fc6eb604c" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
I20251028 14:51:56.965118 2131 raft_consensus.cc:3060] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.965067 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74b5785bff29430c85a5bc0fc6eb604c" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51"
I20251028 14:51:56.965273 2403 raft_consensus.cc:3060] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:56.967149 2131 raft_consensus.cc:2468] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 1.
I20251028 14:51:56.967156 2403 raft_consensus.cc:2468] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 1.
I20251028 14:51:56.967706 2194 leader_election.cc:304] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:56.968046 2475 raft_consensus.cc:2804] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:56.968281 2475 raft_consensus.cc:697] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 LEADER]: Becoming Leader. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Running, Role: LEADER
I20251028 14:51:56.968585 2475 consensus_queue.cc:237] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:56.970203 2483 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: New leader e55bd2b7d0a74b9bbaef40478491d5ca. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:56.970559 2483 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:56.971125 2496 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:51:56.971455 2496 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:51:56.971714 1970 catalog_manager.cc:5649] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } health_report { overall_health: UNKNOWN } } }
I20251028 14:51:56.990978 2471 consensus_queue.cc:1048] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.001798 2471 consensus_queue.cc:1048] T 64cf82de43d447a7ab38ff8ccf253d81 P 73cf821b8d8e4a708d6e6de2516be3c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.032769 2403 raft_consensus.cc:1275] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Refusing update from remote peer e55bd2b7d0a74b9bbaef40478491d5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:57.032920 2131 raft_consensus.cc:1275] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Refusing update from remote peer e55bd2b7d0a74b9bbaef40478491d5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:57.033550 2475 consensus_queue.cc:1048] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.033859 2483 consensus_queue.cc:1048] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.041824 2471 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: New leader e55bd2b7d0a74b9bbaef40478491d5ca. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.042061 2471 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.044334 2503 mvcc.cc:204] Tried to move back new op lower bound from 7215772127357341696 to 7215772127105474560. Current Snapshot: MvccSnapshot[applied={T|T < 7215772127357341696}]
I20251028 14:51:57.049743 2471 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.050007 2471 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.052559 2472 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: New leader e55bd2b7d0a74b9bbaef40478491d5ca. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.052811 2472 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.060755 2472 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.061003 2472 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.061007 2475 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.061496 2475 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:57.064504 2483 tablet_replica.cc:442] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.064792 2483 tablet_replica.cc:445] T 74b5785bff29430c85a5bc0fc6eb604c P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:57.109619 2131 raft_consensus.cc:1275] T 311bcdeaa5d4466db7a06af60ba0cafd P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Refusing update from remote peer e55bd2b7d0a74b9bbaef40478491d5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:57.109642 2403 raft_consensus.cc:1275] T 311bcdeaa5d4466db7a06af60ba0cafd P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Refusing update from remote peer e55bd2b7d0a74b9bbaef40478491d5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:57.110313 2483 consensus_queue.cc:1048] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.110644 2515 consensus_queue.cc:1048] T 311bcdeaa5d4466db7a06af60ba0cafd P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.116456 2499 mvcc.cc:204] Tried to move back new op lower bound from 7215772127674073088 to 7215772125859885056. Current Snapshot: MvccSnapshot[applied={T|T < 7215772127674073088}]
I20251028 14:51:57.387637 1972 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41718:
table { table_name: "kudu_system.kudu_transactions" } alter_schema_steps { type: ADD_RANGE_PARTITION add_range_partition { range_bounds { rows: "<redacted>""\006\001\003\000\000\000\000\000\000\000\007\001\006\000\000\000\000\000\000\000" indirect_data: "<redacted>""" } } } schema { columns { name: "txn_id" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "entry_type" type: INT8 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "identifier" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "metadata" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: false
W20251028 14:51:57.393678 1972 catalog_manager.cc:5101] could not convert tablet's 74b5785bff29430c85a5bc0fc6eb604c schema from PB: OK
W20251028 14:51:57.394217 1972 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68d5098fb8654b72b9b576d8465a65b9 has no consensus state
I20251028 14:51:57.399889 2110 tablet_service.cc:1505] Processing CreateTablet for tablet 68d5098fb8654b72b9b576d8465a65b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 3 <= VALUES < 6
I20251028 14:51:57.400584 2110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68d5098fb8654b72b9b576d8465a65b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:57.400632 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 68d5098fb8654b72b9b576d8465a65b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 3 <= VALUES < 6
I20251028 14:51:57.400825 2383 tablet_service.cc:1505] Processing CreateTablet for tablet 68d5098fb8654b72b9b576d8465a65b9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 3 <= VALUES < 6
I20251028 14:51:57.401121 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68d5098fb8654b72b9b576d8465a65b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:57.401309 2383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68d5098fb8654b72b9b576d8465a65b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:57.415071 2525 tablet_bootstrap.cc:492] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: Bootstrap starting.
I20251028 14:51:57.419570 2525 tablet_bootstrap.cc:654] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:57.421545 2526 tablet_bootstrap.cc:492] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: Bootstrap starting.
I20251028 14:51:57.423465 2526 tablet_bootstrap.cc:654] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:57.425001 2524 tablet_bootstrap.cc:492] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: Bootstrap starting.
I20251028 14:51:57.427147 2525 tablet_bootstrap.cc:492] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: No bootstrap required, opened a new log
I20251028 14:51:57.427364 2524 tablet_bootstrap.cc:654] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:57.427814 2525 ts_tablet_manager.cc:1403] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20251028 14:51:57.431252 2525 raft_consensus.cc:359] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.431728 2525 raft_consensus.cc:385] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:57.431897 2525 raft_consensus.cc:740] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Initialized, Role: FOLLOWER
I20251028 14:51:57.432304 2525 consensus_queue.cc:260] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.432885 2515 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.433105 2515 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.434573 2525 ts_tablet_manager.cc:1434] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251028 14:51:57.436303 2526 tablet_bootstrap.cc:492] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: No bootstrap required, opened a new log
I20251028 14:51:57.436501 2526 ts_tablet_manager.cc:1403] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20251028 14:51:57.437629 2526 raft_consensus.cc:359] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.437839 2526 raft_consensus.cc:385] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:57.437935 2526 raft_consensus.cc:740] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Initialized, Role: FOLLOWER
I20251028 14:51:57.438310 2526 consensus_queue.cc:260] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.438709 2472 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.438870 2472 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.438895 1972 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13] alter complete (version 0)
I20251028 14:51:57.439966 2524 tablet_bootstrap.cc:492] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: No bootstrap required, opened a new log
I20251028 14:51:57.440140 2524 ts_tablet_manager.cc:1403] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20251028 14:51:57.441149 2524 raft_consensus.cc:359] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.441408 2524 raft_consensus.cc:385] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:57.441507 2524 raft_consensus.cc:740] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cf821b8d8e4a708d6e6de2516be3c1, State: Initialized, Role: FOLLOWER
I20251028 14:51:57.441784 2524 consensus_queue.cc:260] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.442127 2526 ts_tablet_manager.cc:1434] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251028 14:51:57.442170 2471 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.442374 2471 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.443367 2524 ts_tablet_manager.cc:1434] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20251028 14:51:57.447467 1957 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 68d5098fb8654b72b9b576d8465a65b9 (table kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13], current schema version=0) failed: Not found: failed to reset TS proxy: No leader found: Tablet 68d5098fb8654b72b9b576d8465a65b9 consensus state has no leader
I20251028 14:51:57.706351 2515 raft_consensus.cc:493] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:57.706643 2515 raft_consensus.cc:515] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.707406 2515 leader_election.cc:290] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965)
I20251028 14:51:57.708472 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68d5098fb8654b72b9b576d8465a65b9" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" is_pre_election: true
I20251028 14:51:57.708483 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68d5098fb8654b72b9b576d8465a65b9" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51" is_pre_election: true
I20251028 14:51:57.708750 2131 raft_consensus.cc:2468] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 0.
I20251028 14:51:57.708889 2403 raft_consensus.cc:2468] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 0.
I20251028 14:51:57.709538 2194 leader_election.cc:304] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:57.710150 2515 raft_consensus.cc:2804] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:57.710323 2515 raft_consensus.cc:493] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:57.710603 2515 raft_consensus.cc:3060] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:57.713243 2515 raft_consensus.cc:515] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.714236 2515 leader_election.cc:290] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 election: Requested vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), 05e78f2c00064213b0e0102302749a51 (127.0.104.3:39965)
I20251028 14:51:57.715219 2403 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68d5098fb8654b72b9b576d8465a65b9" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05e78f2c00064213b0e0102302749a51"
I20251028 14:51:57.715248 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68d5098fb8654b72b9b576d8465a65b9" candidate_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
I20251028 14:51:57.715654 2131 raft_consensus.cc:3060] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:57.715713 2403 raft_consensus.cc:3060] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:57.718226 2403 raft_consensus.cc:2468] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 1.
I20251028 14:51:57.719004 2194 leader_election.cc:304] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:57.719429 2515 raft_consensus.cc:2804] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:57.719662 2515 raft_consensus.cc:697] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 LEADER]: Becoming Leader. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Running, Role: LEADER
I20251028 14:51:57.719849 2131 raft_consensus.cc:2468] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e55bd2b7d0a74b9bbaef40478491d5ca in term 1.
I20251028 14:51:57.720093 2515 consensus_queue.cc:237] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.722072 2475 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: New leader e55bd2b7d0a74b9bbaef40478491d5ca. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.722311 2475 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:57.722996 2531 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:51:57.723285 2531 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:51:57.723856 1972 catalog_manager.cc:5649] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca reported cstate change: term changed from 0 to 1, leader changed from <none> to e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } health_report { overall_health: UNKNOWN } } }
I20251028 14:51:57.733582 2403 raft_consensus.cc:1275] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Refusing update from remote peer e55bd2b7d0a74b9bbaef40478491d5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:57.733753 2131 raft_consensus.cc:1275] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Refusing update from remote peer e55bd2b7d0a74b9bbaef40478491d5ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:57.734222 2475 consensus_queue.cc:1048] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.734635 2515 consensus_queue.cc:1048] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:57.738457 2471 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: New leader e55bd2b7d0a74b9bbaef40478491d5ca. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.738559 2472 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: New leader e55bd2b7d0a74b9bbaef40478491d5ca. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.738696 2471 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.738757 2472 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.740271 2519 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.740512 2519 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:57.742502 2515 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.742779 2515 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:57.743176 2472 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.743386 2472 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.747062 2471 tablet_replica.cc:442] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.747345 2471 tablet_replica.cc:445] T 68d5098fb8654b72b9b576d8465a65b9 P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.970193 1970 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41718:
table { table_name: "kudu_system.kudu_transactions" } alter_schema_steps { type: ADD_RANGE_PARTITION add_range_partition { range_bounds { rows: "<redacted>""\006\001\006\000\000\000\000\000\000\000\007\001\t\000\000\000\000\000\000\000" indirect_data: "<redacted>""" } } } schema { columns { name: "txn_id" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "entry_type" type: INT8 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "identifier" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "metadata" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: false
W20251028 14:51:57.975944 1970 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84ef2d955f4a4905a554711cbc9e5e6c has no consensus state
I20251028 14:51:57.982352 2110 tablet_service.cc:1505] Processing CreateTablet for tablet 84ef2d955f4a4905a554711cbc9e5e6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 6 <= VALUES < 9
I20251028 14:51:57.982566 2242 tablet_service.cc:1505] Processing CreateTablet for tablet 84ef2d955f4a4905a554711cbc9e5e6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 6 <= VALUES < 9
I20251028 14:51:57.983017 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ef2d955f4a4905a554711cbc9e5e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:57.983021 2110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ef2d955f4a4905a554711cbc9e5e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:57.983076 2383 tablet_service.cc:1505] Processing CreateTablet for tablet 84ef2d955f4a4905a554711cbc9e5e6c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 6 <= VALUES < 9
I20251028 14:51:57.983552 2383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ef2d955f4a4905a554711cbc9e5e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:57.988647 2537 tablet_bootstrap.cc:492] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: Bootstrap starting.
I20251028 14:51:57.990605 2538 tablet_bootstrap.cc:492] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: Bootstrap starting.
I20251028 14:51:57.990758 2537 tablet_bootstrap.cc:654] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:57.991029 2539 tablet_bootstrap.cc:492] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: Bootstrap starting.
I20251028 14:51:57.992456 2538 tablet_bootstrap.cc:654] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:57.992671 2539 tablet_bootstrap.cc:654] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:57.993393 2537 tablet_bootstrap.cc:492] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: No bootstrap required, opened a new log
I20251028 14:51:57.993706 2537 ts_tablet_manager.cc:1403] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:51:57.994563 2537 raft_consensus.cc:359] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.994771 2537 raft_consensus.cc:385] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:57.994872 2538 tablet_bootstrap.cc:492] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: No bootstrap required, opened a new log
I20251028 14:51:57.994869 2537 raft_consensus.cc:740] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Initialized, Role: FOLLOWER
I20251028 14:51:57.994920 2539 tablet_bootstrap.cc:492] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: No bootstrap required, opened a new log
I20251028 14:51:57.995026 2538 ts_tablet_manager.cc:1403] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:51:57.995061 2539 ts_tablet_manager.cc:1403] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251028 14:51:57.995108 2537 consensus_queue.cc:260] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.995534 2515 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.995682 2537 ts_tablet_manager.cc:1434] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:51:57.995725 2515 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.995770 2538 raft_consensus.cc:359] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.995841 2539 raft_consensus.cc:359] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.995940 2538 raft_consensus.cc:385] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:57.996008 2539 raft_consensus.cc:385] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:57.996024 2538 raft_consensus.cc:740] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cf821b8d8e4a708d6e6de2516be3c1, State: Initialized, Role: FOLLOWER
I20251028 14:51:57.996097 2539 raft_consensus.cc:740] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Initialized, Role: FOLLOWER
I20251028 14:51:57.996254 2538 consensus_queue.cc:260] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.996335 2539 consensus_queue.cc:260] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:57.996599 2471 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.996735 2471 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:57.996858 2539 ts_tablet_manager.cc:1434] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251028 14:51:57.997128 2538 ts_tablet_manager.cc:1434] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:51:57.997154 2472 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:57.997385 2472 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.001418 1972 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13] alter complete (version 0)
I20251028 14:51:58.052938 2472 raft_consensus.cc:493] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:58.053171 2472 raft_consensus.cc:515] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.054785 2472 leader_election.cc:290] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2:40751)
I20251028 14:51:58.065086 2267 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ef2d955f4a4905a554711cbc9e5e6c" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" is_pre_election: true
I20251028 14:51:58.065097 2130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ef2d955f4a4905a554711cbc9e5e6c" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" is_pre_election: true
I20251028 14:51:58.065384 2130 raft_consensus.cc:2468] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 0.
I20251028 14:51:58.065412 2267 raft_consensus.cc:2468] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 0.
I20251028 14:51:58.066087 2330 leader_election.cc:304] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:58.066807 2472 raft_consensus.cc:2804] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:58.066969 2472 raft_consensus.cc:493] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:58.067102 2472 raft_consensus.cc:3060] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:58.069520 2472 raft_consensus.cc:515] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.070286 2472 leader_election.cc:290] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 election: Requested vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2:40751)
I20251028 14:51:58.070991 2267 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ef2d955f4a4905a554711cbc9e5e6c" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca"
I20251028 14:51:58.071147 2130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84ef2d955f4a4905a554711cbc9e5e6c" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
I20251028 14:51:58.071223 2267 raft_consensus.cc:3060] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:58.071406 2130 raft_consensus.cc:3060] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:58.073719 2130 raft_consensus.cc:2468] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 1.
I20251028 14:51:58.073729 2267 raft_consensus.cc:2468] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 1.
I20251028 14:51:58.074332 2329 leader_election.cc:304] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:58.074757 2472 raft_consensus.cc:2804] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:58.075259 2472 raft_consensus.cc:697] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [term 1 LEADER]: Becoming Leader. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Running, Role: LEADER
I20251028 14:51:58.076097 2472 consensus_queue.cc:237] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.078495 2547 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: New leader 05e78f2c00064213b0e0102302749a51. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.078832 2547 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:58.079666 2549 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:51:58.079934 2549 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:51:58.081092 1971 catalog_manager.cc:5649] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e78f2c00064213b0e0102302749a51 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } health_report { overall_health: HEALTHY } } }
I20251028 14:51:58.106763 2130 raft_consensus.cc:1275] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Refusing update from remote peer 05e78f2c00064213b0e0102302749a51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:58.106788 2267 raft_consensus.cc:1275] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Refusing update from remote peer 05e78f2c00064213b0e0102302749a51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:58.107954 2472 consensus_queue.cc:1048] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:58.108333 2547 consensus_queue.cc:1048] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:58.111544 2515 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: New leader 05e78f2c00064213b0e0102302749a51. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.111846 2515 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.112677 2471 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: New leader 05e78f2c00064213b0e0102302749a51. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.112977 2471 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.114434 2547 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.114655 2547 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:58.116103 2500 mvcc.cc:204] Tried to move back new op lower bound from 7215772131756322816 to 7215772131645759488. Current Snapshot: MvccSnapshot[applied={T|T < 7215772131756322816 or (T in {7215772131756322816})}]
I20251028 14:51:58.116348 2515 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.116593 2515 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.122781 2472 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.123034 2472 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:58.124807 2471 tablet_replica.cc:442] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.125077 2471 tablet_replica.cc:445] T 84ef2d955f4a4905a554711cbc9e5e6c P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.404063 1971 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:41718:
table { table_name: "kudu_system.kudu_transactions" } alter_schema_steps { type: ADD_RANGE_PARTITION add_range_partition { range_bounds { rows: "<redacted>""\006\001\t\000\000\000\000\000\000\000\007\001\014\000\000\000\000\000\000\000" indirect_data: "<redacted>""" } } } schema { columns { name: "txn_id" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "entry_type" type: INT8 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "identifier" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "metadata" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: false
W20251028 14:51:58.409322 1971 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cd25218896534f689d214cb683479d95 has no consensus state
I20251028 14:51:58.413918 2110 tablet_service.cc:1505] Processing CreateTablet for tablet cd25218896534f689d214cb683479d95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 9 <= VALUES < 12
I20251028 14:51:58.414469 2110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd25218896534f689d214cb683479d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:58.414928 2382 tablet_service.cc:1505] Processing CreateTablet for tablet cd25218896534f689d214cb683479d95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 9 <= VALUES < 12
I20251028 14:51:58.414968 2242 tablet_service.cc:1505] Processing CreateTablet for tablet cd25218896534f689d214cb683479d95 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13]), partition=RANGE (txn_id) PARTITION 9 <= VALUES < 12
I20251028 14:51:58.415445 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd25218896534f689d214cb683479d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:58.415737 2382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd25218896534f689d214cb683479d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:51:58.421475 2538 tablet_bootstrap.cc:492] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: Bootstrap starting.
I20251028 14:51:58.422197 2537 tablet_bootstrap.cc:492] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: Bootstrap starting.
I20251028 14:51:58.423501 2538 tablet_bootstrap.cc:654] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:58.423990 2537 tablet_bootstrap.cc:654] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:58.424185 2539 tablet_bootstrap.cc:492] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: Bootstrap starting.
I20251028 14:51:58.425940 2539 tablet_bootstrap.cc:654] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: Neither blocks nor log segments found. Creating new log.
I20251028 14:51:58.426478 2537 tablet_bootstrap.cc:492] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: No bootstrap required, opened a new log
I20251028 14:51:58.426692 2537 ts_tablet_manager.cc:1403] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251028 14:51:58.426913 2538 tablet_bootstrap.cc:492] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: No bootstrap required, opened a new log
I20251028 14:51:58.427122 2538 ts_tablet_manager.cc:1403] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20251028 14:51:58.427623 2537 raft_consensus.cc:359] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.427853 2537 raft_consensus.cc:385] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:58.427933 2537 raft_consensus.cc:740] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55bd2b7d0a74b9bbaef40478491d5ca, State: Initialized, Role: FOLLOWER
I20251028 14:51:58.427901 2538 raft_consensus.cc:359] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.428062 2538 raft_consensus.cc:385] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:58.428136 2538 raft_consensus.cc:740] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73cf821b8d8e4a708d6e6de2516be3c1, State: Initialized, Role: FOLLOWER
I20251028 14:51:58.428207 2537 consensus_queue.cc:260] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.428364 2538 consensus_queue.cc:260] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.428673 2483 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.428794 2537 ts_tablet_manager.cc:1434] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 14:51:58.428874 2483 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.428912 2538 ts_tablet_manager.cc:1434] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 14:51:58.429353 2471 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.429852 2471 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.434281 1971 catalog_manager.cc:6194] T 00000000000000000000000000000000 P 5c6e50781c034dd7ac5eb474140d1b0c: kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13] alter complete (version 0)
I20251028 14:51:58.438110 2539 tablet_bootstrap.cc:492] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: No bootstrap required, opened a new log
I20251028 14:51:58.438437 2539 ts_tablet_manager.cc:1403] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251028 14:51:58.439392 2539 raft_consensus.cc:359] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.439594 2539 raft_consensus.cc:385] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:51:58.439652 2539 raft_consensus.cc:740] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Initialized, Role: FOLLOWER
I20251028 14:51:58.439922 2539 consensus_queue.cc:260] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.440441 2539 ts_tablet_manager.cc:1434] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 14:51:58.440368 2472 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.440601 2472 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: FOLLOWER
W20251028 14:51:58.468396 1957 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cd25218896534f689d214cb683479d95 (table kudu_system.kudu_transactions [id=e9052916e71a48698dcb344510ae7b13], current schema version=0) failed: Not found: failed to reset TS proxy: No leader found: Tablet cd25218896534f689d214cb683479d95 consensus state has no leader
I20251028 14:51:58.609534 2472 raft_consensus.cc:493] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:51:58.609862 2472 raft_consensus.cc:515] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.610795 2472 leader_election.cc:290] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2:40751)
I20251028 14:51:58.611537 2267 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd25218896534f689d214cb683479d95" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" is_pre_election: true
I20251028 14:51:58.611683 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd25218896534f689d214cb683479d95" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" is_pre_election: true
I20251028 14:51:58.611883 2131 raft_consensus.cc:2468] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 0.
I20251028 14:51:58.611893 2267 raft_consensus.cc:2468] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 0.
I20251028 14:51:58.612488 2330 leader_election.cc:304] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, 73cf821b8d8e4a708d6e6de2516be3c1, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:58.612891 2472 raft_consensus.cc:2804] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:51:58.613073 2472 raft_consensus.cc:493] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:51:58.613183 2472 raft_consensus.cc:3060] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:58.617293 2472 raft_consensus.cc:515] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.618130 2472 leader_election.cc:290] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 election: Requested vote from peers 73cf821b8d8e4a708d6e6de2516be3c1 (127.0.104.1:37033), e55bd2b7d0a74b9bbaef40478491d5ca (127.0.104.2:40751)
I20251028 14:51:58.618818 2267 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd25218896534f689d214cb683479d95" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca"
I20251028 14:51:58.618825 2131 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd25218896534f689d214cb683479d95" candidate_uuid: "05e78f2c00064213b0e0102302749a51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73cf821b8d8e4a708d6e6de2516be3c1"
I20251028 14:51:58.619047 2131 raft_consensus.cc:3060] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:58.619064 2267 raft_consensus.cc:3060] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:51:58.622196 2131 raft_consensus.cc:2468] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 1.
I20251028 14:51:58.622188 2267 raft_consensus.cc:2468] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05e78f2c00064213b0e0102302749a51 in term 1.
I20251028 14:51:58.622777 2329 leader_election.cc:304] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05e78f2c00064213b0e0102302749a51, e55bd2b7d0a74b9bbaef40478491d5ca; no voters:
I20251028 14:51:58.623180 2472 raft_consensus.cc:2804] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:51:58.623843 2472 raft_consensus.cc:697] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [term 1 LEADER]: Becoming Leader. State: Replica: 05e78f2c00064213b0e0102302749a51, State: Running, Role: LEADER
I20251028 14:51:58.624266 2472 consensus_queue.cc:237] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } }
I20251028 14:51:58.626669 2547 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: New leader 05e78f2c00064213b0e0102302749a51. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.627005 2547 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:58.628968 1970 catalog_manager.cc:5649] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05e78f2c00064213b0e0102302749a51 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } health_report { overall_health: HEALTHY } } }
I20251028 14:51:58.639828 2571 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:51:58.640030 2571 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:51:58.688263 2131 raft_consensus.cc:1275] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1 [term 1 FOLLOWER]: Refusing update from remote peer 05e78f2c00064213b0e0102302749a51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:58.688330 2267 raft_consensus.cc:1275] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca [term 1 FOLLOWER]: Refusing update from remote peer 05e78f2c00064213b0e0102302749a51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:51:58.688892 2472 consensus_queue.cc:1048] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:58.689146 2547 consensus_queue.cc:1048] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:51:58.691932 2483 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: New leader 05e78f2c00064213b0e0102302749a51. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.692144 2483 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.692572 2471 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: New leader 05e78f2c00064213b0e0102302749a51. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.692795 2471 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.695036 2547 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.695282 2547 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:58.695926 2558 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.696154 2558 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P 05e78f2c00064213b0e0102302749a51: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:51:58.698372 2483 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.698611 2483 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P e55bd2b7d0a74b9bbaef40478491d5ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.702358 2471 tablet_replica.cc:442] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "05e78f2c00064213b0e0102302749a51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "73cf821b8d8e4a708d6e6de2516be3c1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37033 } } peers { permanent_uuid: "e55bd2b7d0a74b9bbaef40478491d5ca" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40751 } } peers { permanent_uuid: "05e78f2c00064213b0e0102302749a51" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39965 } } }
I20251028 14:51:58.702633 2471 tablet_replica.cc:445] T cd25218896534f689d214cb683479d95 P 73cf821b8d8e4a708d6e6de2516be3c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:51:58.789695 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 2041
I20251028 14:51:58.791157 2171 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:58.949450 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 2041
I20251028 14:51:58.968127 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 2178
W20251028 14:51:58.968459 2194 connection.cc:537] client connection to 127.0.104.1:37033 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 14:51:58.968750 2194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:51:58.969408 2307 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:59.117040 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 2178
W20251028 14:51:59.137926 2331 connection.cc:537] server connection from 127.0.104.2:51971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:51:59.138531 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 2314
I20251028 14:51:59.139635 2443 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:59.270126 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 2314
I20251028 14:51:59.290262 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 1941
I20251028 14:51:59.291903 2012 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:51:59.413534 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 1941
2025-10-28T14:51:59Z chronyd exiting
[ OK ] TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds (5258 ms)
[ RUN ] TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit
2025-10-28T14:51:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:51:59Z Disabled control of system clock
I20251028 14:51:59.482465 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35629
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35629
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20251028 14:51:59.870265 2591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:51:59.870659 2591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:51:59.870752 2591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:51:59.882191 2591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:51:59.882551 2591 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:51:59.882746 2591 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20251028 14:51:59.882799 2591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:51:59.882866 2591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:51:59.882922 2591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:51:59.896051 2591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35629
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:51:59.899185 2591 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:51:59.902657 2591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:51:59.916265 2596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:59.916314 2597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:51:59.916647 2599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:51:59.919224 2591 server_base.cc:1047] running on GCE node
I20251028 14:51:59.921399 2591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:51:59.923904 2591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:51:59.925562 2591 hybrid_clock.cc:648] HybridClock initialized: now 1761663119925180 us; error 81 us; skew 500 ppm
I20251028 14:51:59.926368 2591 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:51:59.930294 2591 webserver.cc:492] Webserver started at http://127.0.104.62:32833/ using document root <none> and password file <none>
I20251028 14:51:59.931252 2591 fs_manager.cc:362] Metadata directory not provided
I20251028 14:51:59.931381 2591 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:51:59.931748 2591 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:51:59.934640 2591 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "c3a928ae5df6475b8df465ef42359138"
format_stamp: "Formatted at 2025-10-28 14:51:59 on dist-test-slave-0v9m"
I20251028 14:51:59.935595 2591 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "c3a928ae5df6475b8df465ef42359138"
format_stamp: "Formatted at 2025-10-28 14:51:59 on dist-test-slave-0v9m"
I20251028 14:51:59.944625 2591 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.003s
I20251028 14:51:59.950562 2605 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:51:59.953135 2591 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20251028 14:51:59.953796 2591 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "c3a928ae5df6475b8df465ef42359138"
format_stamp: "Formatted at 2025-10-28 14:51:59 on dist-test-slave-0v9m"
I20251028 14:51:59.954109 2591 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:51:59.989885 2591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:51:59.990881 2591 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:51:59.991155 2591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:00.020211 2591 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35629
I20251028 14:52:00.020232 2666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35629 every 8 connection(s)
I20251028 14:52:00.021910 2591 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:52:00.027460 2667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:00.028028 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 2591
I20251028 14:52:00.028280 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:52:00.044581 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138: Bootstrap starting.
I20251028 14:52:00.049197 2667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:00.050879 2667 log.cc:826] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:00.054999 2667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138: No bootstrap required, opened a new log
I20251028 14:52:00.061200 2667 raft_consensus.cc:359] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a928ae5df6475b8df465ef42359138" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35629 } }
I20251028 14:52:00.061754 2667 raft_consensus.cc:385] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:00.061872 2667 raft_consensus.cc:740] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3a928ae5df6475b8df465ef42359138, State: Initialized, Role: FOLLOWER
I20251028 14:52:00.062659 2667 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a928ae5df6475b8df465ef42359138" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35629 } }
I20251028 14:52:00.062929 2667 raft_consensus.cc:399] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:00.063072 2667 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:00.063292 2667 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:00.065882 2667 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a928ae5df6475b8df465ef42359138" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35629 } }
I20251028 14:52:00.066566 2667 leader_election.cc:304] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3a928ae5df6475b8df465ef42359138; no voters:
I20251028 14:52:00.067415 2667 leader_election.cc:290] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:00.067587 2672 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:00.068281 2672 raft_consensus.cc:697] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [term 1 LEADER]: Becoming Leader. State: Replica: c3a928ae5df6475b8df465ef42359138, State: Running, Role: LEADER
I20251028 14:52:00.068864 2672 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a928ae5df6475b8df465ef42359138" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35629 } }
I20251028 14:52:00.069684 2667 sys_catalog.cc:565] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:00.073429 2673 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3a928ae5df6475b8df465ef42359138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a928ae5df6475b8df465ef42359138" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35629 } } }
I20251028 14:52:00.072474 2674 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3a928ae5df6475b8df465ef42359138. Latest consensus state: current_term: 1 leader_uuid: "c3a928ae5df6475b8df465ef42359138" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3a928ae5df6475b8df465ef42359138" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35629 } } }
I20251028 14:52:00.075924 2673 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:00.075989 2674 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:00.079124 2680 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:00.086598 2680 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:00.102223 2680 catalog_manager.cc:1357] Generated new cluster ID: 82317a497cf1471dbc2510e1455ece46
I20251028 14:52:00.102418 2680 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:00.144094 2680 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:00.146224 2680 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:00.158877 2680 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c3a928ae5df6475b8df465ef42359138: Generated new TSK 0
I20251028 14:52:00.159940 2680 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:00.167104 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35629
--builtin_ntp_servers=127.0.104.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:52:00.178231 2621 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35620:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:00.187682 2667 master.cc:464] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20251028 14:52:00.619587 2696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:00.619923 2696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:00.620014 2696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:00.628573 2696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:00.628738 2696 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:00.628818 2696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:52:00.640498 2696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:00.642935 2696 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:00.645983 2696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:00.661197 2702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:00.661175 2701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:00.662521 2704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:00.663156 2696 server_base.cc:1047] running on GCE node
I20251028 14:52:00.664034 2696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:00.665618 2696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:00.666945 2696 hybrid_clock.cc:648] HybridClock initialized: now 1761663120666820 us; error 92 us; skew 500 ppm
I20251028 14:52:00.667397 2696 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:00.671144 2696 webserver.cc:492] Webserver started at http://127.0.104.1:38529/ using document root <none> and password file <none>
I20251028 14:52:00.672040 2696 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:00.672173 2696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:00.672582 2696 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:00.675452 2696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5"
format_stamp: "Formatted at 2025-10-28 14:52:00 on dist-test-slave-0v9m"
I20251028 14:52:00.676347 2696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5"
format_stamp: "Formatted at 2025-10-28 14:52:00 on dist-test-slave-0v9m"
I20251028 14:52:00.685189 2696 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20251028 14:52:00.690647 2710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:00.692911 2696 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20251028 14:52:00.693334 2696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5"
format_stamp: "Formatted at 2025-10-28 14:52:00 on dist-test-slave-0v9m"
I20251028 14:52:00.693676 2696 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:00.734568 2696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:00.735641 2696 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:00.735963 2696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:00.740962 2696 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:00.741175 2696 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:00.741356 2696 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:00.741647 2696 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:00.810678 2696 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45561
I20251028 14:52:00.810747 2830 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45561 every 8 connection(s)
I20251028 14:52:00.812274 2696 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:52:00.815362 2831 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35629
I20251028 14:52:00.815737 2831 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:00.816658 2831 heartbeater.cc:507] Master 127.0.104.62:35629 requested a full tablet report, sending...
I20251028 14:52:00.819097 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 2696
I20251028 14:52:00.819295 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:52:00.820519 2621 ts_manager.cc:194] Registered new tserver with Master: 1e8ed4da8c7b4dd5984b352a9b2a8bb5 (127.0.104.1:45561)
I20251028 14:52:00.823047 2621 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:60589
I20251028 14:52:00.825785 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35629
--builtin_ntp_servers=127.0.104.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:52:01.234270 2621 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35624:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:01.259716 2833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:01.260071 2833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:01.260205 2833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:01.269819 2833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:01.269985 2833 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:01.270076 2833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:52:01.281725 2833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:01.284720 2833 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:01.288167 2833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:01.301312 2844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:01.301676 2843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:01.303526 2846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:01.304752 2833 server_base.cc:1047] running on GCE node
I20251028 14:52:01.305936 2833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:01.307570 2833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:01.309096 2833 hybrid_clock.cc:648] HybridClock initialized: now 1761663121308940 us; error 138 us; skew 500 ppm
I20251028 14:52:01.309631 2833 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:01.313159 2833 webserver.cc:492] Webserver started at http://127.0.104.2:39821/ using document root <none> and password file <none>
I20251028 14:52:01.314378 2833 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:01.314615 2833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:01.315089 2833 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:01.318672 2833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "0e677f8ce6f74f4181365b20a4e47059"
format_stamp: "Formatted at 2025-10-28 14:52:01 on dist-test-slave-0v9m"
I20251028 14:52:01.320067 2833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "0e677f8ce6f74f4181365b20a4e47059"
format_stamp: "Formatted at 2025-10-28 14:52:01 on dist-test-slave-0v9m"
I20251028 14:52:01.327841 2833 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20251028 14:52:01.333168 2852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:01.334988 2833 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20251028 14:52:01.335206 2833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "0e677f8ce6f74f4181365b20a4e47059"
format_stamp: "Formatted at 2025-10-28 14:52:01 on dist-test-slave-0v9m"
I20251028 14:52:01.335438 2833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:01.366267 2833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:01.367235 2833 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:01.367548 2833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:01.371119 2833 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:01.371251 2833 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:01.371387 2833 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:01.371518 2833 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251028 14:52:01.426512 2833 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44699
I20251028 14:52:01.426697 2971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44699 every 8 connection(s)
I20251028 14:52:01.428263 2833 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:52:01.431082 2972 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35629
I20251028 14:52:01.431502 2972 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:01.431649 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 2833
I20251028 14:52:01.431854 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:52:01.432744 2972 heartbeater.cc:507] Master 127.0.104.62:35629 requested a full tablet report, sending...
I20251028 14:52:01.435211 2621 ts_manager.cc:194] Registered new tserver with Master: 0e677f8ce6f74f4181365b20a4e47059 (127.0.104.2:44699)
I20251028 14:52:01.435804 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35629
--builtin_ntp_servers=127.0.104.20:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:52:01.436367 2621 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:46757
W20251028 14:52:01.818122 2974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:01.818464 2974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:01.818594 2974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:01.827754 2974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:01.827919 2974 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:01.828029 2974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:52:01.828683 2831 heartbeater.cc:499] Master 127.0.104.62:35629 was elected leader, sending a full tablet report...
I20251028 14:52:01.839244 2974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35629
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:01.841029 2974 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:01.843024 2974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:01.852874 2979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:01.852957 2980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:01.854318 2982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:01.854746 2974 server_base.cc:1047] running on GCE node
I20251028 14:52:01.855949 2974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:01.857064 2974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:01.858350 2974 hybrid_clock.cc:648] HybridClock initialized: now 1761663121858161 us; error 169 us; skew 500 ppm
I20251028 14:52:01.858697 2974 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:01.861231 2974 webserver.cc:492] Webserver started at http://127.0.104.3:34281/ using document root <none> and password file <none>
I20251028 14:52:01.862133 2974 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:01.862264 2974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:01.862635 2974 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:01.865409 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "5ee2602f76604e6c9d18004bf00d0a09"
format_stamp: "Formatted at 2025-10-28 14:52:01 on dist-test-slave-0v9m"
I20251028 14:52:01.866248 2974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "5ee2602f76604e6c9d18004bf00d0a09"
format_stamp: "Formatted at 2025-10-28 14:52:01 on dist-test-slave-0v9m"
I20251028 14:52:01.872380 2974 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251028 14:52:01.876371 2988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:01.878213 2974 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:01.878391 2974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "5ee2602f76604e6c9d18004bf00d0a09"
format_stamp: "Formatted at 2025-10-28 14:52:01 on dist-test-slave-0v9m"
I20251028 14:52:01.878575 2974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:01.902032 2974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:01.902922 2974 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:01.903180 2974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:01.906144 2974 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:01.906250 2974 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:01.906383 2974 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:01.906469 2974 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:01.949975 2974 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36079
I20251028 14:52:01.950126 3108 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36079 every 8 connection(s)
I20251028 14:52:01.951565 2974 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:52:01.954454 3109 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35629
I20251028 14:52:01.954816 3109 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:01.955675 3109 heartbeater.cc:507] Master 127.0.104.62:35629 requested a full tablet report, sending...
I20251028 14:52:01.957468 2621 ts_manager.cc:194] Registered new tserver with Master: 5ee2602f76604e6c9d18004bf00d0a09 (127.0.104.3:36079)
I20251028 14:52:01.958300 2621 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:47973
I20251028 14:52:01.959394 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 2974
I20251028 14:52:01.959587 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:52:01.963356 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:52:01.980062 2621 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35630:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:52:01.981143 2621 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:02.013505 3043 tablet_service.cc:1505] Processing CreateTablet for tablet 8ac6d022c23f4feb8c01f353d78cf2b3 (DEFAULT_TABLE table=txn_write_ops_test [id=49427226faea4794a3e3b72fb9e9be62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:02.014253 2907 tablet_service.cc:1505] Processing CreateTablet for tablet 8ac6d022c23f4feb8c01f353d78cf2b3 (DEFAULT_TABLE table=txn_write_ops_test [id=49427226faea4794a3e3b72fb9e9be62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:02.014463 2765 tablet_service.cc:1505] Processing CreateTablet for tablet 1b5a2f87e3a14bb68b115a6e40dc75bd (DEFAULT_TABLE table=txn_write_ops_test [id=49427226faea4794a3e3b72fb9e9be62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:02.015355 3043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac6d022c23f4feb8c01f353d78cf2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.015551 2765 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5a2f87e3a14bb68b115a6e40dc75bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.015727 2907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac6d022c23f4feb8c01f353d78cf2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.014073 2906 tablet_service.cc:1505] Processing CreateTablet for tablet 1b5a2f87e3a14bb68b115a6e40dc75bd (DEFAULT_TABLE table=txn_write_ops_test [id=49427226faea4794a3e3b72fb9e9be62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:02.016453 2906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5a2f87e3a14bb68b115a6e40dc75bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.014076 3042 tablet_service.cc:1505] Processing CreateTablet for tablet 1b5a2f87e3a14bb68b115a6e40dc75bd (DEFAULT_TABLE table=txn_write_ops_test [id=49427226faea4794a3e3b72fb9e9be62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:02.017493 3042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b5a2f87e3a14bb68b115a6e40dc75bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.013504 2766 tablet_service.cc:1505] Processing CreateTablet for tablet 8ac6d022c23f4feb8c01f353d78cf2b3 (DEFAULT_TABLE table=txn_write_ops_test [id=49427226faea4794a3e3b72fb9e9be62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:02.020545 2766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac6d022c23f4feb8c01f353d78cf2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.040046 3128 tablet_bootstrap.cc:492] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Bootstrap starting.
I20251028 14:52:02.041841 3126 tablet_bootstrap.cc:492] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09: Bootstrap starting.
I20251028 14:52:02.046922 3128 tablet_bootstrap.cc:654] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.048557 3127 tablet_bootstrap.cc:492] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059: Bootstrap starting.
I20251028 14:52:02.049522 3128 log.cc:826] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:02.051066 3126 tablet_bootstrap.cc:654] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.052373 3127 tablet_bootstrap.cc:654] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.052726 3126 log.cc:826] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:02.053102 3128 tablet_bootstrap.cc:492] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: No bootstrap required, opened a new log
I20251028 14:52:02.053506 3128 ts_tablet_manager.cc:1403] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20251028 14:52:02.053610 3127 log.cc:826] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:02.056232 3127 tablet_bootstrap.cc:492] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059: No bootstrap required, opened a new log
I20251028 14:52:02.056550 3127 ts_tablet_manager.cc:1403] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20251028 14:52:02.057063 3126 tablet_bootstrap.cc:492] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09: No bootstrap required, opened a new log
I20251028 14:52:02.057438 3126 ts_tablet_manager.cc:1403] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.003s
I20251028 14:52:02.061319 3128 raft_consensus.cc:359] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.061985 3127 raft_consensus.cc:359] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.062395 3128 raft_consensus.cc:385] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.062343 3127 raft_consensus.cc:385] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.062536 3127 raft_consensus.cc:740] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e677f8ce6f74f4181365b20a4e47059, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.062521 3128 raft_consensus.cc:740] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8ed4da8c7b4dd5984b352a9b2a8bb5, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.063269 3127 consensus_queue.cc:260] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.063467 3128 consensus_queue.cc:260] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.064374 2972 heartbeater.cc:499] Master 127.0.104.62:35629 was elected leader, sending a full tablet report...
I20251028 14:52:02.065084 3127 ts_tablet_manager.cc:1434] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20251028 14:52:02.064946 3126 raft_consensus.cc:359] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.065726 3126 raft_consensus.cc:385] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.065760 3127 tablet_bootstrap.cc:492] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059: Bootstrap starting.
I20251028 14:52:02.065909 3126 raft_consensus.cc:740] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2602f76604e6c9d18004bf00d0a09, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.067505 3128 ts_tablet_manager.cc:1434] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20251028 14:52:02.067660 3127 tablet_bootstrap.cc:654] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.068166 3128 tablet_bootstrap.cc:492] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Bootstrap starting.
I20251028 14:52:02.070238 3128 tablet_bootstrap.cc:654] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.071344 3126 consensus_queue.cc:260] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.080199 3126 ts_tablet_manager.cc:1434] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09: Time spent starting tablet: real 0.023s user 0.007s sys 0.009s
I20251028 14:52:02.080060 3128 tablet_bootstrap.cc:492] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: No bootstrap required, opened a new log
I20251028 14:52:02.080832 3109 heartbeater.cc:499] Master 127.0.104.62:35629 was elected leader, sending a full tablet report...
I20251028 14:52:02.080930 3128 ts_tablet_manager.cc:1403] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20251028 14:52:02.080562 3133 raft_consensus.cc:493] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:02.081732 3133 raft_consensus.cc:515] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.081959 3128 raft_consensus.cc:359] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.082098 3128 raft_consensus.cc:385] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.082165 3128 raft_consensus.cc:740] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8ed4da8c7b4dd5984b352a9b2a8bb5, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.082374 3127 tablet_bootstrap.cc:492] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059: No bootstrap required, opened a new log
I20251028 14:52:02.082392 3128 consensus_queue.cc:260] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.082592 3127 ts_tablet_manager.cc:1403] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.014s
I20251028 14:52:02.083487 3127 raft_consensus.cc:359] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.083716 3127 raft_consensus.cc:385] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.083832 3127 raft_consensus.cc:740] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e677f8ce6f74f4181365b20a4e47059, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.084076 3133 leader_election.cc:290] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ee2602f76604e6c9d18004bf00d0a09 (127.0.104.3:36079), 0e677f8ce6f74f4181365b20a4e47059 (127.0.104.2:44699)
I20251028 14:52:02.084134 3127 consensus_queue.cc:260] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.084662 3127 ts_tablet_manager.cc:1434] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251028 14:52:02.090693 3126 tablet_bootstrap.cc:492] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09: Bootstrap starting.
I20251028 14:52:02.093150 3126 tablet_bootstrap.cc:654] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.095125 3128 ts_tablet_manager.cc:1434] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20251028 14:52:02.095688 3126 tablet_bootstrap.cc:492] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09: No bootstrap required, opened a new log
I20251028 14:52:02.095881 3126 ts_tablet_manager.cc:1403] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20251028 14:52:02.096836 3126 raft_consensus.cc:359] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.097153 3126 raft_consensus.cc:385] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.097373 3126 raft_consensus.cc:740] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2602f76604e6c9d18004bf00d0a09, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.098129 3064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ac6d022c23f4feb8c01f353d78cf2b3" candidate_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2602f76604e6c9d18004bf00d0a09" is_pre_election: true
I20251028 14:52:02.098366 3126 consensus_queue.cc:260] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.098682 3064 raft_consensus.cc:2468] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e8ed4da8c7b4dd5984b352a9b2a8bb5 in term 0.
I20251028 14:52:02.099377 3126 ts_tablet_manager.cc:1434] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251028 14:52:02.099843 2712 leader_election.cc:304] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8ed4da8c7b4dd5984b352a9b2a8bb5, 5ee2602f76604e6c9d18004bf00d0a09; no voters:
I20251028 14:52:02.100281 3137 raft_consensus.cc:2804] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:02.100428 3137 raft_consensus.cc:493] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:02.100569 3137 raft_consensus.cc:3060] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.100723 2927 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ac6d022c23f4feb8c01f353d78cf2b3" candidate_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e677f8ce6f74f4181365b20a4e47059" is_pre_election: true
I20251028 14:52:02.101406 2927 raft_consensus.cc:2468] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e8ed4da8c7b4dd5984b352a9b2a8bb5 in term 0.
I20251028 14:52:02.103893 3137 raft_consensus.cc:515] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.104553 3137 leader_election.cc:290] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [CANDIDATE]: Term 1 election: Requested vote from peers 5ee2602f76604e6c9d18004bf00d0a09 (127.0.104.3:36079), 0e677f8ce6f74f4181365b20a4e47059 (127.0.104.2:44699)
I20251028 14:52:02.105098 3064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ac6d022c23f4feb8c01f353d78cf2b3" candidate_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2602f76604e6c9d18004bf00d0a09"
I20251028 14:52:02.105365 3064 raft_consensus.cc:3060] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.105351 2927 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8ac6d022c23f4feb8c01f353d78cf2b3" candidate_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e677f8ce6f74f4181365b20a4e47059"
I20251028 14:52:02.105549 2927 raft_consensus.cc:3060] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.107754 3064 raft_consensus.cc:2468] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e8ed4da8c7b4dd5984b352a9b2a8bb5 in term 1.
I20251028 14:52:02.107828 2927 raft_consensus.cc:2468] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e8ed4da8c7b4dd5984b352a9b2a8bb5 in term 1.
I20251028 14:52:02.108369 2712 leader_election.cc:304] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e677f8ce6f74f4181365b20a4e47059, 1e8ed4da8c7b4dd5984b352a9b2a8bb5, 5ee2602f76604e6c9d18004bf00d0a09; no voters:
I20251028 14:52:02.108695 3137 raft_consensus.cc:2804] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:02.108912 3137 raft_consensus.cc:697] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 1e8ed4da8c7b4dd5984b352a9b2a8bb5, State: Running, Role: LEADER
I20251028 14:52:02.109465 3137 consensus_queue.cc:237] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } }
I20251028 14:52:02.114459 2621 catalog_manager.cc:5649] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e8ed4da8c7b4dd5984b352a9b2a8bb5 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:02.122932 3136 raft_consensus.cc:493] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:02.123236 3136 raft_consensus.cc:515] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.125490 3136 leader_election.cc:290] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e8ed4da8c7b4dd5984b352a9b2a8bb5 (127.0.104.1:45561), 0e677f8ce6f74f4181365b20a4e47059 (127.0.104.2:44699)
I20251028 14:52:02.132754 2927 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5a2f87e3a14bb68b115a6e40dc75bd" candidate_uuid: "5ee2602f76604e6c9d18004bf00d0a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e677f8ce6f74f4181365b20a4e47059" is_pre_election: true
I20251028 14:52:02.133064 2927 raft_consensus.cc:2468] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee2602f76604e6c9d18004bf00d0a09 in term 0.
I20251028 14:52:02.133518 2786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5a2f87e3a14bb68b115a6e40dc75bd" candidate_uuid: "5ee2602f76604e6c9d18004bf00d0a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" is_pre_election: true
I20251028 14:52:02.133764 2989 leader_election.cc:304] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e677f8ce6f74f4181365b20a4e47059, 5ee2602f76604e6c9d18004bf00d0a09; no voters:
I20251028 14:52:02.133967 2786 raft_consensus.cc:2468] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ee2602f76604e6c9d18004bf00d0a09 in term 0.
I20251028 14:52:02.134282 3136 raft_consensus.cc:2804] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:02.134424 3136 raft_consensus.cc:493] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:02.134523 3136 raft_consensus.cc:3060] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.137624 3136 raft_consensus.cc:515] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.138449 3136 leader_election.cc:290] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [CANDIDATE]: Term 1 election: Requested vote from peers 1e8ed4da8c7b4dd5984b352a9b2a8bb5 (127.0.104.1:45561), 0e677f8ce6f74f4181365b20a4e47059 (127.0.104.2:44699)
I20251028 14:52:02.139303 2786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5a2f87e3a14bb68b115a6e40dc75bd" candidate_uuid: "5ee2602f76604e6c9d18004bf00d0a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5"
I20251028 14:52:02.139513 2786 raft_consensus.cc:3060] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.139621 2927 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1b5a2f87e3a14bb68b115a6e40dc75bd" candidate_uuid: "5ee2602f76604e6c9d18004bf00d0a09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e677f8ce6f74f4181365b20a4e47059"
I20251028 14:52:02.140025 2927 raft_consensus.cc:3060] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.142042 2786 raft_consensus.cc:2468] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee2602f76604e6c9d18004bf00d0a09 in term 1.
I20251028 14:52:02.142791 2992 leader_election.cc:304] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e8ed4da8c7b4dd5984b352a9b2a8bb5, 5ee2602f76604e6c9d18004bf00d0a09; no voters:
I20251028 14:52:02.142915 2927 raft_consensus.cc:2468] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ee2602f76604e6c9d18004bf00d0a09 in term 1.
I20251028 14:52:02.143206 3136 raft_consensus.cc:2804] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:02.143966 3136 raft_consensus.cc:697] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [term 1 LEADER]: Becoming Leader. State: Replica: 5ee2602f76604e6c9d18004bf00d0a09, State: Running, Role: LEADER
I20251028 14:52:02.144611 3136 consensus_queue.cc:237] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.149816 2621 catalog_manager.cc:5649] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ee2602f76604e6c9d18004bf00d0a09 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "5ee2602f76604e6c9d18004bf00d0a09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:02.180738 2973 tablet.cc:2378] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:02.198648 3113 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
W20251028 14:52:02.203593 3110 tablet.cc:2378] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:52:02.252700 2621 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35638:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:02.253495 2621 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:02.261686 2766 tablet_service.cc:1505] Processing CreateTablet for tablet 726a862d0718411aba91320cf6ebb881 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c732127ca08a45dd9024e91e99411cf3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:02.261916 2906 tablet_service.cc:1505] Processing CreateTablet for tablet 726a862d0718411aba91320cf6ebb881 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c732127ca08a45dd9024e91e99411cf3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:02.262012 3042 tablet_service.cc:1505] Processing CreateTablet for tablet 726a862d0718411aba91320cf6ebb881 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c732127ca08a45dd9024e91e99411cf3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:02.262821 2906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726a862d0718411aba91320cf6ebb881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.262825 2766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726a862d0718411aba91320cf6ebb881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.262940 3042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 726a862d0718411aba91320cf6ebb881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:02.269685 3126 tablet_bootstrap.cc:492] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: Bootstrap starting.
I20251028 14:52:02.270584 3128 tablet_bootstrap.cc:492] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Bootstrap starting.
I20251028 14:52:02.271615 3126 tablet_bootstrap.cc:654] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.271776 3127 tablet_bootstrap.cc:492] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: Bootstrap starting.
I20251028 14:52:02.272598 3128 tablet_bootstrap.cc:654] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.273602 3127 tablet_bootstrap.cc:654] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:02.273830 3126 tablet_bootstrap.cc:492] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: No bootstrap required, opened a new log
I20251028 14:52:02.274004 3126 ts_tablet_manager.cc:1403] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20251028 14:52:02.274780 3128 tablet_bootstrap.cc:492] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: No bootstrap required, opened a new log
I20251028 14:52:02.274828 3126 raft_consensus.cc:359] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.275045 3128 ts_tablet_manager.cc:1403] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20251028 14:52:02.275161 3126 raft_consensus.cc:385] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.275291 3126 raft_consensus.cc:740] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ee2602f76604e6c9d18004bf00d0a09, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.275580 3126 consensus_queue.cc:260] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.275872 3127 tablet_bootstrap.cc:492] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: No bootstrap required, opened a new log
I20251028 14:52:02.275871 3128 raft_consensus.cc:359] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.276096 3127 ts_tablet_manager.cc:1403] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251028 14:52:02.276110 3128 raft_consensus.cc:385] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.275974 3136 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.276283 3128 raft_consensus.cc:740] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e8ed4da8c7b4dd5984b352a9b2a8bb5, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.276499 3136 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:02.276597 3128 consensus_queue.cc:260] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.276873 3127 raft_consensus.cc:359] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.277076 3127 raft_consensus.cc:385] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:02.277156 3127 raft_consensus.cc:740] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e677f8ce6f74f4181365b20a4e47059, State: Initialized, Role: FOLLOWER
I20251028 14:52:02.277088 3137 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.277484 3127 consensus_queue.cc:260] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.277201 3128 ts_tablet_manager.cc:1434] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:52:02.277688 3137 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:02.277858 3132 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.278132 3132 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:02.278324 3126 ts_tablet_manager.cc:1434] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:02.278998 3127 ts_tablet_manager.cc:1434] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251028 14:52:02.315660 2832 tablet.cc:2378] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:52:02.327994 3132 raft_consensus.cc:493] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:02.328228 3132 raft_consensus.cc:515] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.330179 3132 leader_election.cc:290] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e8ed4da8c7b4dd5984b352a9b2a8bb5 (127.0.104.1:45561), 5ee2602f76604e6c9d18004bf00d0a09 (127.0.104.3:36079)
I20251028 14:52:02.338392 2786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726a862d0718411aba91320cf6ebb881" candidate_uuid: "0e677f8ce6f74f4181365b20a4e47059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" is_pre_election: true
I20251028 14:52:02.338425 3064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726a862d0718411aba91320cf6ebb881" candidate_uuid: "0e677f8ce6f74f4181365b20a4e47059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2602f76604e6c9d18004bf00d0a09" is_pre_election: true
I20251028 14:52:02.338655 2786 raft_consensus.cc:2468] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e677f8ce6f74f4181365b20a4e47059 in term 0.
I20251028 14:52:02.338657 3064 raft_consensus.cc:2468] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e677f8ce6f74f4181365b20a4e47059 in term 0.
I20251028 14:52:02.339175 2854 leader_election.cc:304] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e677f8ce6f74f4181365b20a4e47059, 5ee2602f76604e6c9d18004bf00d0a09; no voters:
I20251028 14:52:02.339689 3132 raft_consensus.cc:2804] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:02.339838 3132 raft_consensus.cc:493] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:02.339931 3132 raft_consensus.cc:3060] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.342260 3132 raft_consensus.cc:515] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.342993 3132 leader_election.cc:290] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [CANDIDATE]: Term 1 election: Requested vote from peers 1e8ed4da8c7b4dd5984b352a9b2a8bb5 (127.0.104.1:45561), 5ee2602f76604e6c9d18004bf00d0a09 (127.0.104.3:36079)
I20251028 14:52:02.343538 2786 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726a862d0718411aba91320cf6ebb881" candidate_uuid: "0e677f8ce6f74f4181365b20a4e47059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5"
I20251028 14:52:02.343597 3064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "726a862d0718411aba91320cf6ebb881" candidate_uuid: "0e677f8ce6f74f4181365b20a4e47059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ee2602f76604e6c9d18004bf00d0a09"
I20251028 14:52:02.343773 2786 raft_consensus.cc:3060] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.343799 3064 raft_consensus.cc:3060] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:02.345873 2786 raft_consensus.cc:2468] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e677f8ce6f74f4181365b20a4e47059 in term 1.
I20251028 14:52:02.345875 3064 raft_consensus.cc:2468] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e677f8ce6f74f4181365b20a4e47059 in term 1.
I20251028 14:52:02.346413 2856 leader_election.cc:304] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e677f8ce6f74f4181365b20a4e47059, 1e8ed4da8c7b4dd5984b352a9b2a8bb5; no voters:
I20251028 14:52:02.346751 3132 raft_consensus.cc:2804] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:02.347198 3132 raft_consensus.cc:697] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [term 1 LEADER]: Becoming Leader. State: Replica: 0e677f8ce6f74f4181365b20a4e47059, State: Running, Role: LEADER
I20251028 14:52:02.347718 3132 consensus_queue.cc:237] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } }
I20251028 14:52:02.351433 3156 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: TxnStatusTablet state changed. Reason: New leader 0e677f8ce6f74f4181365b20a4e47059. Latest consensus state: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.351756 3156 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:02.352488 3158 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:02.352846 3158 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:02.353574 2620 catalog_manager.cc:5649] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e677f8ce6f74f4181365b20a4e47059 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:02.437587 2786 raft_consensus.cc:1275] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 1 FOLLOWER]: Refusing update from remote peer 0e677f8ce6f74f4181365b20a4e47059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:02.437575 3064 raft_consensus.cc:1275] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09 [term 1 FOLLOWER]: Refusing update from remote peer 0e677f8ce6f74f4181365b20a4e47059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:02.438877 3156 consensus_queue.cc:1048] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:02.439205 3132 consensus_queue.cc:1048] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:02.455881 3136 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: TxnStatusTablet state changed. Reason: New leader 0e677f8ce6f74f4181365b20a4e47059. Latest consensus state: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.456193 3136 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:02.464699 3161 mvcc.cc:204] Tried to move back new op lower bound from 7215772149490073600 to 7215772149140738048. Current Snapshot: MvccSnapshot[applied={T|T < 7215772149490073600}]
I20251028 14:52:02.468994 3137 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: TxnStatusTablet state changed. Reason: New leader 0e677f8ce6f74f4181365b20a4e47059. Latest consensus state: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.469419 3137 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:02.470319 3156 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.470613 3156 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:02.472209 3136 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.472479 3136 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 5ee2602f76604e6c9d18004bf00d0a09: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:02.483675 3163 mvcc.cc:204] Tried to move back new op lower bound from 7215772149490073600 to 7215772149140738048. Current Snapshot: MvccSnapshot[applied={T|T < 7215772149490073600}]
I20251028 14:52:02.487277 3137 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.487586 3137 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:02.489460 3169 tablet_replica.cc:442] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e677f8ce6f74f4181365b20a4e47059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 } } peers { permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 } } peers { permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 } } }
I20251028 14:52:02.490907 3169 tablet_replica.cc:445] T 726a862d0718411aba91320cf6ebb881 P 0e677f8ce6f74f4181365b20a4e47059: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:02.531080 3064 raft_consensus.cc:1275] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 5ee2602f76604e6c9d18004bf00d0a09 [term 1 FOLLOWER]: Refusing update from remote peer 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:02.531363 2927 raft_consensus.cc:1275] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 0e677f8ce6f74f4181365b20a4e47059 [term 1 FOLLOWER]: Refusing update from remote peer 1e8ed4da8c7b4dd5984b352a9b2a8bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:02.532070 3137 consensus_queue.cc:1048] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ee2602f76604e6c9d18004bf00d0a09" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:02.532423 3133 consensus_queue.cc:1048] T 8ac6d022c23f4feb8c01f353d78cf2b3 P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:02.540729 3162 mvcc.cc:204] Tried to move back new op lower bound from 7215772149877559296 to 7215772148164710400. Current Snapshot: MvccSnapshot[applied={T|T < 7215772149877559296}]
I20251028 14:52:02.596227 2786 raft_consensus.cc:1275] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 1e8ed4da8c7b4dd5984b352a9b2a8bb5 [term 1 FOLLOWER]: Refusing update from remote peer 5ee2602f76604e6c9d18004bf00d0a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:02.596382 2927 raft_consensus.cc:1275] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 0e677f8ce6f74f4181365b20a4e47059 [term 1 FOLLOWER]: Refusing update from remote peer 5ee2602f76604e6c9d18004bf00d0a09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:02.597028 3143 consensus_queue.cc:1048] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e8ed4da8c7b4dd5984b352a9b2a8bb5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:02.597401 3136 consensus_queue.cc:1048] T 1b5a2f87e3a14bb68b115a6e40dc75bd P 5ee2602f76604e6c9d18004bf00d0a09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e677f8ce6f74f4181365b20a4e47059" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:09.916015 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 2696
I20251028 14:52:09.917665 2826 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:10.051643 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 2696
I20251028 14:52:10.077862 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 2833
I20251028 14:52:10.079195 2967 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:10.247529 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 2833
I20251028 14:52:10.272709 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 2974
I20251028 14:52:10.274453 3104 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:10.462555 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 2974
I20251028 14:52:10.486436 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 2591
I20251028 14:52:10.487656 2662 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:10.605022 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 2591
2025-10-28T14:52:10Z chronyd exiting
[ OK ] TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit (11193 ms)
[ RUN ] TxnWriteOpsITest.FrequentElections
2025-10-28T14:52:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:52:10Z Disabled control of system clock
I20251028 14:52:10.680569 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40211
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40211
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20251028 14:52:11.047091 3233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:11.047472 3233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:11.047591 3233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:11.056334 3233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:52:11.056469 3233 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:52:11.056540 3233 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20251028 14:52:11.056597 3233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:11.056680 3233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:52:11.056746 3233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:52:11.068349 3233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40211
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:11.070271 3233 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:11.072274 3233 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:11.082718 3238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:11.082990 3239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:11.083645 3241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:11.084048 3233 server_base.cc:1047] running on GCE node
I20251028 14:52:11.084951 3233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:11.087184 3233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:11.088531 3233 hybrid_clock.cc:648] HybridClock initialized: now 1761663131088464 us; error 61 us; skew 500 ppm
I20251028 14:52:11.088958 3233 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:11.091742 3233 webserver.cc:492] Webserver started at http://127.0.104.62:41299/ using document root <none> and password file <none>
I20251028 14:52:11.092566 3233 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:11.092684 3233 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:11.093032 3233 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:11.095619 3233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "b45ce2b5e914497baffcfa962207e0e7"
format_stamp: "Formatted at 2025-10-28 14:52:11 on dist-test-slave-0v9m"
I20251028 14:52:11.096606 3233 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "b45ce2b5e914497baffcfa962207e0e7"
format_stamp: "Formatted at 2025-10-28 14:52:11 on dist-test-slave-0v9m"
I20251028 14:52:11.106071 3233 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.000s
I20251028 14:52:11.111977 3247 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:11.114455 3233 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20251028 14:52:11.114691 3233 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "b45ce2b5e914497baffcfa962207e0e7"
format_stamp: "Formatted at 2025-10-28 14:52:11 on dist-test-slave-0v9m"
I20251028 14:52:11.114917 3233 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:11.148705 3233 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:11.149690 3233 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:11.149956 3233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:11.175072 3233 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40211
I20251028 14:52:11.175092 3308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40211 every 8 connection(s)
I20251028 14:52:11.176705 3233 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:52:11.182837 3309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:11.183817 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 3233
I20251028 14:52:11.184051 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:52:11.197974 3309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7: Bootstrap starting.
I20251028 14:52:11.201618 3309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:11.203186 3309 log.cc:826] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:11.206799 3309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7: No bootstrap required, opened a new log
I20251028 14:52:11.214748 3309 raft_consensus.cc:359] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ce2b5e914497baffcfa962207e0e7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40211 } }
I20251028 14:52:11.215176 3309 raft_consensus.cc:385] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:11.215282 3309 raft_consensus.cc:740] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b45ce2b5e914497baffcfa962207e0e7, State: Initialized, Role: FOLLOWER
I20251028 14:52:11.215946 3309 consensus_queue.cc:260] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ce2b5e914497baffcfa962207e0e7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40211 } }
I20251028 14:52:11.216171 3309 raft_consensus.cc:399] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:11.216295 3309 raft_consensus.cc:493] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:11.216444 3309 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:11.218880 3309 raft_consensus.cc:515] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ce2b5e914497baffcfa962207e0e7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40211 } }
I20251028 14:52:11.219455 3309 leader_election.cc:304] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b45ce2b5e914497baffcfa962207e0e7; no voters:
I20251028 14:52:11.220168 3309 leader_election.cc:290] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:11.220292 3314 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:11.222436 3309 sys_catalog.cc:565] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:11.225582 3314 raft_consensus.cc:697] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [term 1 LEADER]: Becoming Leader. State: Replica: b45ce2b5e914497baffcfa962207e0e7, State: Running, Role: LEADER
I20251028 14:52:11.226356 3314 consensus_queue.cc:237] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ce2b5e914497baffcfa962207e0e7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40211 } }
I20251028 14:52:11.229700 3315 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b45ce2b5e914497baffcfa962207e0e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ce2b5e914497baffcfa962207e0e7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40211 } } }
I20251028 14:52:11.230368 3316 sys_catalog.cc:455] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b45ce2b5e914497baffcfa962207e0e7. Latest consensus state: current_term: 1 leader_uuid: "b45ce2b5e914497baffcfa962207e0e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b45ce2b5e914497baffcfa962207e0e7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40211 } } }
I20251028 14:52:11.230726 3315 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:11.231060 3316 sys_catalog.cc:458] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7 [sys.catalog]: This master's current role is: LEADER
W20251028 14:52:11.238878 3328 catalog_manager.cc:1568] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251028 14:52:11.239135 3328 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251028 14:52:11.239315 3330 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:11.241245 3330 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:11.250952 3330 catalog_manager.cc:1357] Generated new cluster ID: 462caa521d594f81b196044025fdff0b
I20251028 14:52:11.251099 3330 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:11.267786 3330 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:11.269104 3330 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:11.281320 3330 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b45ce2b5e914497baffcfa962207e0e7: Generated new TSK 0
I20251028 14:52:11.282835 3330 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:11.295369 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40211
--builtin_ntp_servers=127.0.104.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--heartbeat_interval_ms=64
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20251028 14:52:11.295501 3264 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51050:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:11.305027 3309 master.cc:464] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20251028 14:52:11.680222 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:11.680626 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:11.680716 3338 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20251028 14:52:11.680794 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:11.689558 3338 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20251028 14:52:11.689713 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:11.689808 3338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:11.689903 3338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:52:11.699978 3338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true
--heartbeat_interval_ms=64
--tserver_master_addrs=127.0.104.62:40211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:11.701633 3338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:11.703517 3338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:11.713951 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:11.713991 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:11.715319 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:11.715492 3338 server_base.cc:1047] running on GCE node
I20251028 14:52:11.716269 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:11.717482 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:11.718761 3338 hybrid_clock.cc:648] HybridClock initialized: now 1761663131718681 us; error 57 us; skew 500 ppm
I20251028 14:52:11.719107 3338 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:11.721866 3338 webserver.cc:492] Webserver started at http://127.0.104.1:46287/ using document root <none> and password file <none>
I20251028 14:52:11.722751 3338 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:11.722890 3338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:11.723275 3338 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:11.726148 3338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
format_stamp: "Formatted at 2025-10-28 14:52:11 on dist-test-slave-0v9m"
I20251028 14:52:11.727005 3338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
format_stamp: "Formatted at 2025-10-28 14:52:11 on dist-test-slave-0v9m"
I20251028 14:52:11.733198 3338 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20251028 14:52:11.737353 3352 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:11.739100 3338 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:11.739279 3338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
format_stamp: "Formatted at 2025-10-28 14:52:11 on dist-test-slave-0v9m"
I20251028 14:52:11.739470 3338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:11.760644 3338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:11.761596 3338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:11.761858 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:11.764930 3338 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:11.765057 3338 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:11.765225 3338 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:11.765357 3338 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:11.811925 3338 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39177
I20251028 14:52:11.812009 3472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39177 every 8 connection(s)
I20251028 14:52:11.813427 3338 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:52:11.815778 3473 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40211
I20251028 14:52:11.816051 3473 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:11.816854 3473 heartbeater.cc:507] Master 127.0.104.62:40211 requested a full tablet report, sending...
I20251028 14:52:11.819001 3264 ts_manager.cc:194] Registered new tserver with Master: d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177)
I20251028 14:52:11.820647 3264 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:44739
I20251028 14:52:11.823442 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 3338
I20251028 14:52:11.823663 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:52:11.827342 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40211
--builtin_ntp_servers=127.0.104.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--heartbeat_interval_ms=64
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20251028 14:52:11.887807 3473 heartbeater.cc:499] Master 127.0.104.62:40211 was elected leader, sending a full tablet report...
W20251028 14:52:12.175004 3475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:12.175367 3475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:12.175526 3475 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20251028 14:52:12.175669 3475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:12.186054 3475 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20251028 14:52:12.186262 3475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:12.186401 3475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:12.186539 3475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:52:12.198608 3475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true
--heartbeat_interval_ms=64
--tserver_master_addrs=127.0.104.62:40211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:12.200345 3475 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:12.202346 3475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:12.212893 3480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:12.212771 3481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:12.213124 3483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:12.216408 3475 server_base.cc:1047] running on GCE node
I20251028 14:52:12.217303 3475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:12.218523 3475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:12.219856 3475 hybrid_clock.cc:648] HybridClock initialized: now 1761663132219793 us; error 46 us; skew 500 ppm
I20251028 14:52:12.220202 3475 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:12.222884 3475 webserver.cc:492] Webserver started at http://127.0.104.2:42939/ using document root <none> and password file <none>
I20251028 14:52:12.223739 3475 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:12.223889 3475 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:12.224260 3475 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:12.227005 3475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
format_stamp: "Formatted at 2025-10-28 14:52:12 on dist-test-slave-0v9m"
I20251028 14:52:12.227916 3475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
format_stamp: "Formatted at 2025-10-28 14:52:12 on dist-test-slave-0v9m"
I20251028 14:52:12.234957 3475 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20251028 14:52:12.240136 3489 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:12.241884 3475 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251028 14:52:12.242076 3475 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
format_stamp: "Formatted at 2025-10-28 14:52:12 on dist-test-slave-0v9m"
I20251028 14:52:12.242282 3475 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:12.276075 3475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:12.277025 3475 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:12.277335 3475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:12.281081 3475 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:12.281344 3475 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251028 14:52:12.281575 3475 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:12.281749 3475 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:12.342840 3264 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51060:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:12.343421 3475 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41917
I20251028 14:52:12.343504 3613 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41917 every 8 connection(s)
I20251028 14:52:12.344956 3475 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:52:12.347493 3614 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40211
I20251028 14:52:12.347864 3614 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:12.348718 3614 heartbeater.cc:507] Master 127.0.104.62:40211 requested a full tablet report, sending...
I20251028 14:52:12.349339 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 3475
I20251028 14:52:12.349561 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:52:12.350613 3264 ts_manager.cc:194] Registered new tserver with Master: 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:12.351810 3264 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:45691
I20251028 14:52:12.354101 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40211
--builtin_ntp_servers=127.0.104.20:36001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--heartbeat_interval_ms=64
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20251028 14:52:12.419708 3614 heartbeater.cc:499] Master 127.0.104.62:40211 was elected leader, sending a full tablet report...
W20251028 14:52:12.712646 3616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:12.713013 3616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:12.713111 3616 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20251028 14:52:12.713215 3616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:12.721745 3616 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20251028 14:52:12.721907 3616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:12.722029 3616 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:12.722151 3616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:52:12.733021 3616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true
--heartbeat_interval_ms=64
--tserver_master_addrs=127.0.104.62:40211
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:12.735350 3616 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:12.738047 3616 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:12.748766 3624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:12.748857 3622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:12.749117 3621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:12.750214 3616 server_base.cc:1047] running on GCE node
I20251028 14:52:12.750885 3616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:12.752019 3616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:12.753540 3616 hybrid_clock.cc:648] HybridClock initialized: now 1761663132753387 us; error 120 us; skew 500 ppm
I20251028 14:52:12.753861 3616 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:12.756309 3616 webserver.cc:492] Webserver started at http://127.0.104.3:35829/ using document root <none> and password file <none>
I20251028 14:52:12.757093 3616 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:12.757210 3616 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:12.757596 3616 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:12.760110 3616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "d400e7c613e146ab91729d201413b76c"
format_stamp: "Formatted at 2025-10-28 14:52:12 on dist-test-slave-0v9m"
I20251028 14:52:12.760924 3616 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "d400e7c613e146ab91729d201413b76c"
format_stamp: "Formatted at 2025-10-28 14:52:12 on dist-test-slave-0v9m"
I20251028 14:52:12.766901 3616 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20251028 14:52:12.771023 3630 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:12.772648 3616 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:52:12.772879 3616 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "d400e7c613e146ab91729d201413b76c"
format_stamp: "Formatted at 2025-10-28 14:52:12 on dist-test-slave-0v9m"
I20251028 14:52:12.773089 3616 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:12.810236 3616 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:12.811149 3616 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:12.811827 3616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:12.815187 3616 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:12.815398 3616 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251028 14:52:12.815604 3616 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:12.815742 3616 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:12.862839 3616 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:37725
I20251028 14:52:12.862902 3749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:37725 every 8 connection(s)
I20251028 14:52:12.864327 3616 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:52:12.867444 3750 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40211
I20251028 14:52:12.867587 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 3616
I20251028 14:52:12.867745 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:52:12.867863 3750 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:12.868935 3750 heartbeater.cc:507] Master 127.0.104.62:40211 requested a full tablet report, sending...
I20251028 14:52:12.870494 3264 ts_manager.cc:194] Registered new tserver with Master: d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:12.871425 3264 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:51457
I20251028 14:52:12.871464 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:52:12.888352 3263 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51076:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:52:12.889564 3263 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:12.922816 3544 tablet_service.cc:1505] Processing CreateTablet for tablet 253448b9924d42f48cd2cf3e477284c3 (DEFAULT_TABLE table=txn_write_ops_test [id=7e6d6f6bfe4c47508797a972d1f4b311]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:12.924577 3544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253448b9924d42f48cd2cf3e477284c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:12.926034 3543 tablet_service.cc:1505] Processing CreateTablet for tablet c69e5c1a69cf438180b9f398eeaba707 (DEFAULT_TABLE table=txn_write_ops_test [id=7e6d6f6bfe4c47508797a972d1f4b311]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:12.927582 3543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69e5c1a69cf438180b9f398eeaba707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:12.930963 3407 tablet_service.cc:1505] Processing CreateTablet for tablet 253448b9924d42f48cd2cf3e477284c3 (DEFAULT_TABLE table=txn_write_ops_test [id=7e6d6f6bfe4c47508797a972d1f4b311]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:12.932816 3407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253448b9924d42f48cd2cf3e477284c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:12.936108 3680 tablet_service.cc:1505] Processing CreateTablet for tablet 253448b9924d42f48cd2cf3e477284c3 (DEFAULT_TABLE table=txn_write_ops_test [id=7e6d6f6bfe4c47508797a972d1f4b311]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:12.930986 3406 tablet_service.cc:1505] Processing CreateTablet for tablet c69e5c1a69cf438180b9f398eeaba707 (DEFAULT_TABLE table=txn_write_ops_test [id=7e6d6f6bfe4c47508797a972d1f4b311]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:12.937705 3406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69e5c1a69cf438180b9f398eeaba707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:12.937922 3680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253448b9924d42f48cd2cf3e477284c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:12.938516 3679 tablet_service.cc:1505] Processing CreateTablet for tablet c69e5c1a69cf438180b9f398eeaba707 (DEFAULT_TABLE table=txn_write_ops_test [id=7e6d6f6bfe4c47508797a972d1f4b311]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:12.939054 3679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69e5c1a69cf438180b9f398eeaba707. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:12.940652 3750 heartbeater.cc:499] Master 127.0.104.62:40211 was elected leader, sending a full tablet report...
I20251028 14:52:12.949359 3767 tablet_bootstrap.cc:492] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Bootstrap starting.
I20251028 14:52:12.953191 3768 tablet_bootstrap.cc:492] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Bootstrap starting.
I20251028 14:52:12.954264 3767 tablet_bootstrap.cc:654] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:12.955675 3767 log.cc:826] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:12.958767 3768 tablet_bootstrap.cc:654] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:12.959327 3769 tablet_bootstrap.cc:492] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Bootstrap starting.
I20251028 14:52:12.960831 3768 log.cc:826] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:12.962260 3767 tablet_bootstrap.cc:492] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: No bootstrap required, opened a new log
I20251028 14:52:12.962580 3767 ts_tablet_manager.cc:1403] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20251028 14:52:12.963285 3769 tablet_bootstrap.cc:654] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:12.964731 3769 log.cc:826] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:12.965046 3768 tablet_bootstrap.cc:492] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: No bootstrap required, opened a new log
I20251028 14:52:12.965413 3768 ts_tablet_manager.cc:1403] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20251028 14:52:12.968163 3767 raft_consensus.cc:359] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.968410 3769 tablet_bootstrap.cc:492] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: No bootstrap required, opened a new log
I20251028 14:52:12.968530 3767 raft_consensus.cc:385] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:12.968627 3767 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Initialized, Role: FOLLOWER
I20251028 14:52:12.968732 3769 ts_tablet_manager.cc:1403] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20251028 14:52:12.969421 3767 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.974647 3767 ts_tablet_manager.cc:1434] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20251028 14:52:12.973858 3768 raft_consensus.cc:359] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.975169 3767 tablet_bootstrap.cc:492] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Bootstrap starting.
I20251028 14:52:12.975402 3768 raft_consensus.cc:385] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:12.975569 3768 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Initialized, Role: FOLLOWER
I20251028 14:52:12.976158 3769 raft_consensus.cc:359] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.976692 3769 raft_consensus.cc:385] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:12.976809 3769 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Initialized, Role: FOLLOWER
I20251028 14:52:12.976802 3768 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.977082 3767 tablet_bootstrap.cc:654] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:12.977710 3769 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.982441 3769 ts_tablet_manager.cc:1434] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20251028 14:52:12.984475 3769 tablet_bootstrap.cc:492] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: Bootstrap starting.
I20251028 14:52:12.986454 3769 tablet_bootstrap.cc:654] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:12.989405 3768 ts_tablet_manager.cc:1434] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Time spent starting tablet: real 0.024s user 0.018s sys 0.003s
I20251028 14:52:12.990072 3768 tablet_bootstrap.cc:492] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Bootstrap starting.
I20251028 14:52:12.990945 3767 tablet_bootstrap.cc:492] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: No bootstrap required, opened a new log
I20251028 14:52:12.990962 3769 tablet_bootstrap.cc:492] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: No bootstrap required, opened a new log
I20251028 14:52:12.991120 3767 ts_tablet_manager.cc:1403] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20251028 14:52:12.991118 3769 ts_tablet_manager.cc:1403] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20251028 14:52:12.992101 3768 tablet_bootstrap.cc:654] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:12.991997 3767 raft_consensus.cc:359] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.992190 3767 raft_consensus.cc:385] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:12.992136 3769 raft_consensus.cc:359] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.992278 3767 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Initialized, Role: FOLLOWER
I20251028 14:52:12.992376 3769 raft_consensus.cc:385] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:12.992479 3769 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Initialized, Role: FOLLOWER
I20251028 14:52:12.992614 3767 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.992794 3769 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.993202 3767 ts_tablet_manager.cc:1434] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251028 14:52:12.994014 3769 ts_tablet_manager.cc:1434] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251028 14:52:12.996933 3768 tablet_bootstrap.cc:492] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: No bootstrap required, opened a new log
I20251028 14:52:12.997114 3768 ts_tablet_manager.cc:1403] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251028 14:52:12.997942 3768 raft_consensus.cc:359] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.998140 3768 raft_consensus.cc:385] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:12.998232 3768 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Initialized, Role: FOLLOWER
I20251028 14:52:12.998471 3768 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:12.999305 3768 ts_tablet_manager.cc:1434] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251028 14:52:13.066239 3474 tablet.cc:2378] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:13.097304 3615 tablet.cc:2378] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:13.116568 3751 tablet.cc:2378] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:52:13.221760 3773 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:13.222030 3773 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.224418 3773 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.226254 3773 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 1 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.238117 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.238619 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.239279 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.239667 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.240943 3428 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 1.
I20251028 14:52:13.241732 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.242163 3773 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:13.242336 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 1.
I20251028 14:52:13.242697 3773 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.243242 3773 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.245884 3774 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:13.246014 3774 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.248358 3774 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.248513 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.249855 3774 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 1 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.257157 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.257444 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.257452 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.257808 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.259841 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 1.
I20251028 14:52:13.260035 3564 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 1.
I20251028 14:52:13.260599 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.261072 3774 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:13.261732 3774 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.262277 3774 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.267185 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:13.310113 3752 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:52:13.311115 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.318181 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.318554 3428 raft_consensus.cc:476] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 LEADER]: Not starting forced leader election -- already a leader
I20251028 14:52:13.320618 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.320870 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.320979 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:13.323392 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.324227 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 2 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.324764 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.324941 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.324941 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.325019 3564 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 LEADER]: Stepping down as leader of term 1
I20251028 14:52:13.325110 3564 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.325163 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:13.325466 3564 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.326053 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:13.328986 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.329191 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 2.
I20251028 14:52:13.332434 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.332809 3774 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 FOLLOWER]: Leader election won for term 2
I20251028 14:52:13.332983 3774 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.333240 3774 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.333628 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.333942 3564 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.334071 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:13.336557 3564 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.336495 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 1 to 2, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 2 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.337301 3564 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 2 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.337917 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.338150 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:13.338725 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.338917 3564 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.339043 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 FOLLOWER]: Advancing to term 3
I20251028 14:52:13.339315 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.339529 3428 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 LEADER]: Stepping down as leader of term 1
I20251028 14:52:13.339720 3428 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.340145 3428 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.340687 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:13.342149 3564 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.343089 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 2.
I20251028 14:52:13.343575 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.343778 3564 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 3 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.343714 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.343997 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 LEADER]: Stepping down as leader of term 2
I20251028 14:52:13.344053 3773 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 FOLLOWER]: Leader election won for term 2
I20251028 14:52:13.344095 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.344220 3773 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.344393 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.344484 3773 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.344704 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 FOLLOWER]: Advancing to term 3
I20251028 14:52:13.345461 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.345649 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 2 FOLLOWER]: Advancing to term 3
I20251028 14:52:13.351708 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251028 14:52:13.353098 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.353319 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.354859 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 3.
I20251028 14:52:13.355474 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.356055 3781 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 FOLLOWER]: Leader election won for term 3
I20251028 14:52:13.366057 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 1 to 2, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 2 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.367470 3781 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.367941 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.367954 3781 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.368311 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.368458 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 2 FOLLOWER]: Advancing to term 3
I20251028 14:52:13.372290 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.373539 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 2 to 3, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 3 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.387395 3263 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51078:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:13.388402 3263 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:13.388882 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.389117 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 2 FOLLOWER]: Advancing to term 3
I20251028 14:52:13.391633 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.397333 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 3 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.398298 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.398547 3564 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 LEADER]: Stepping down as leader of term 2
I20251028 14:52:13.398640 3564 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.398198 3543 tablet_service.cc:1505] Processing CreateTablet for tablet 5ee80079eb8c42578768d1d87767ed19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=112e62ecca1e47038f4560da31469484]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:13.398979 3564 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.399251 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 2 FOLLOWER]: Advancing to term 3
I20251028 14:52:13.399448 3543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee80079eb8c42578768d1d87767ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:13.398550 3679 tablet_service.cc:1505] Processing CreateTablet for tablet 5ee80079eb8c42578768d1d87767ed19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=112e62ecca1e47038f4560da31469484]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:13.400678 3679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee80079eb8c42578768d1d87767ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:13.402527 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.403407 3407 tablet_service.cc:1505] Processing CreateTablet for tablet 5ee80079eb8c42578768d1d87767ed19 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=112e62ecca1e47038f4560da31469484]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:13.404397 3407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee80079eb8c42578768d1d87767ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:13.405504 3767 tablet_bootstrap.cc:492] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Bootstrap starting.
I20251028 14:52:13.406971 3767 tablet_bootstrap.cc:654] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:13.408214 3769 tablet_bootstrap.cc:492] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: Bootstrap starting.
I20251028 14:52:13.409981 3769 tablet_bootstrap.cc:654] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:13.410346 3767 tablet_bootstrap.cc:492] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: No bootstrap required, opened a new log
I20251028 14:52:13.410514 3767 ts_tablet_manager.cc:1403] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251028 14:52:13.410420 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.410938 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20251028 14:52:13.411674 3767 raft_consensus.cc:359] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.411780 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.411929 3767 raft_consensus.cc:385] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:13.411965 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.411994 3767 raft_consensus.cc:740] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Initialized, Role: FOLLOWER
I20251028 14:52:13.412101 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 3 FOLLOWER]: Advancing to term 4
I20251028 14:52:13.412256 3767 consensus_queue.cc:260] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.412694 3781 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.412953 3781 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:13.414211 3768 tablet_bootstrap.cc:492] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: Bootstrap starting.
I20251028 14:52:13.414846 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.415493 3767 ts_tablet_manager.cc:1434] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251028 14:52:13.415494 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 4 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.416126 3768 tablet_bootstrap.cc:654] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:13.416333 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.416368 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.416592 3564 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 LEADER]: Stepping down as leader of term 3
I20251028 14:52:13.416679 3564 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.417013 3564 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.417151 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.417347 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 FOLLOWER]: Advancing to term 4
I20251028 14:52:13.417313 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 4 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.417411 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.417512 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 3 FOLLOWER]: Advancing to term 4
I20251028 14:52:13.417548 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 3 FOLLOWER]: Advancing to term 4
I20251028 14:52:13.420109 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.420867 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 4 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.421295 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.421432 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.421598 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 3 FOLLOWER]: Advancing to term 4
I20251028 14:52:13.421620 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 3 FOLLOWER]: Advancing to term 4
I20251028 14:52:13.421856 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.422133 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.424068 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 4.
I20251028 14:52:13.420800 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.424264 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20251028 14:52:13.424580 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.424610 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.424739 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 FOLLOWER]: Advancing to term 5
I20251028 14:52:13.424964 3774 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 FOLLOWER]: Leader election won for term 4
I20251028 14:52:13.424942 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.425102 3774 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.425469 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20251028 14:52:13.425908 3769 tablet_bootstrap.cc:492] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: No bootstrap required, opened a new log
I20251028 14:52:13.425825 3774 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.426081 3769 ts_tablet_manager.cc:1403] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20251028 14:52:13.426833 3769 raft_consensus.cc:359] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.427028 3769 raft_consensus.cc:385] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:13.427099 3769 raft_consensus.cc:740] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Initialized, Role: FOLLOWER
I20251028 14:52:13.427337 3769 consensus_queue.cc:260] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.427819 3769 ts_tablet_manager.cc:1434] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 14:52:13.428440 3776 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.428754 3776 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:13.429174 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.430255 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 5 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.430490 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 2 to 4, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 4 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.430718 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.430806 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.430941 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 4 FOLLOWER]: Advancing to term 5
I20251028 14:52:13.431603 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.431833 3564 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.431951 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 4 FOLLOWER]: Advancing to term 5
I20251028 14:52:13.432045 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.432667 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 4 FOLLOWER]: Advancing to term 5
I20251028 14:52:13.433750 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20251028 14:52:13.435006 3564 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.435745 3564 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 5 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.436468 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 5.
I20251028 14:52:13.436662 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.436928 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.436975 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 4 FOLLOWER]: Advancing to term 5
I20251028 14:52:13.437412 3774 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 FOLLOWER]: Leader election won for term 5
I20251028 14:52:13.437625 3774 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.437927 3774 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.438493 3768 tablet_bootstrap.cc:492] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: No bootstrap required, opened a new log
I20251028 14:52:13.438673 3768 ts_tablet_manager.cc:1403] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20251028 14:52:13.439442 3768 raft_consensus.cc:359] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.439636 3768 raft_consensus.cc:385] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:13.439715 3768 raft_consensus.cc:740] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Initialized, Role: FOLLOWER
I20251028 14:52:13.439924 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 5.
I20251028 14:52:13.439992 3768 consensus_queue.cc:260] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.440352 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.440465 3768 ts_tablet_manager.cc:1434] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 14:52:13.440371 3774 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.440591 3774 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:13.436905 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.440682 3781 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 FOLLOWER]: Leader election won for term 5
I20251028 14:52:13.440829 3781 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.441061 3781 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.441161 3428 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 LEADER]: Stepping down as leader of term 4
I20251028 14:52:13.441242 3428 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.441066 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 3 to 5, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 5 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.441530 3428 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.441752 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 4 FOLLOWER]: Advancing to term 5
I20251028 14:52:13.442723 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.442894 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.443011 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 FOLLOWER]: Advancing to term 6
I20251028 14:52:13.443858 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 4 to 5, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 5 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.445066 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20251028 14:52:13.445152 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.445819 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 6 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.446393 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.446556 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.446771 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 LEADER]: Stepping down as leader of term 5
I20251028 14:52:13.446868 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.447132 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.447208 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.447388 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 FOLLOWER]: Advancing to term 6
I20251028 14:52:13.447422 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.447563 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 5 FOLLOWER]: Advancing to term 6
I20251028 14:52:13.450342 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20251028 14:52:13.450366 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.450939 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.451025 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 6 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.451157 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 5 FOLLOWER]: Advancing to term 6
I20251028 14:52:13.451398 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 6 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.451592 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 5 FOLLOWER]: Advancing to term 6
I20251028 14:52:13.451653 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 6 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.452055 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 LEADER]: Stepping down as leader of term 5
I20251028 14:52:13.452154 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.452287 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.452482 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.452795 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 5 FOLLOWER]: Advancing to term 6
I20251028 14:52:13.454061 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.454197 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 6.
I20251028 14:52:13.454470 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.454602 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 6 FOLLOWER]: Advancing to term 7
I20251028 14:52:13.454749 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.455062 3781 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 FOLLOWER]: Leader election won for term 6
I20251028 14:52:13.455200 3781 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.455482 3781 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.455883 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.456749 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.457053 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20251028 14:52:13.457268 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.457864 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 5 to 6, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 6 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.458846 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 7 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.459342 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 7 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.459427 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.459532 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 6 FOLLOWER]: Advancing to term 7
I20251028 14:52:13.460700 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.461141 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.461298 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 6 FOLLOWER]: Advancing to term 7
I20251028 14:52:13.462497 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.462949 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 7 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.463178 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 LEADER]: Stepping down as leader of term 6
I20251028 14:52:13.463274 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.463578 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.463850 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 FOLLOWER]: Advancing to term 7
I20251028 14:52:13.464460 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.465145 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 7 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.465723 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.465765 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.465919 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 6 FOLLOWER]: Advancing to term 7
I20251028 14:52:13.465945 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 6 FOLLOWER]: Advancing to term 7
I20251028 14:52:13.466615 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.466949 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.467141 3427 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.467144 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.467239 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 FOLLOWER]: Advancing to term 8
I20251028 14:52:13.467600 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20251028 14:52:13.468847 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 7.
I20251028 14:52:13.468951 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20251028 14:52:13.469286 3427 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.469398 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.469740 3774 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 FOLLOWER]: Leader election won for term 7
I20251028 14:52:13.469918 3774 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.470180 3774 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.470481 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.470639 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 7 FOLLOWER]: Advancing to term 8
I20251028 14:52:13.471565 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.471773 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 7 FOLLOWER]: Advancing to term 8
I20251028 14:52:13.472693 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 5 to 7, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 7 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.472932 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 8.
I20251028 14:52:13.473342 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.473590 3774 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 FOLLOWER]: Leader election won for term 8
I20251028 14:52:13.473707 3774 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.473893 3774 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.474540 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20251028 14:52:13.470252 3427 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 8 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.475272 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.476061 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.476281 3564 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.476398 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 7 FOLLOWER]: Advancing to term 8
I20251028 14:52:13.477774 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 6 to 8, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 8 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.478511 3564 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.479117 3564 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 8 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.479619 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.479807 3427 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 LEADER]: Stepping down as leader of term 7
I20251028 14:52:13.479905 3427 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.480160 3427 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.480422 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 7 FOLLOWER]: Advancing to term 8
I20251028 14:52:13.480612 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.480803 3564 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.480906 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 FOLLOWER]: Advancing to term 9
I20251028 14:52:13.483505 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20251028 14:52:13.483642 3564 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.484000 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.484218 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 7 FOLLOWER]: Advancing to term 8
I20251028 14:52:13.484311 3564 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 9 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.484755 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.484867 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.485029 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 LEADER]: Stepping down as leader of term 8
I20251028 14:52:13.485051 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.485217 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 8 FOLLOWER]: Advancing to term 9
I20251028 14:52:13.485239 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.485996 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.486289 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 FOLLOWER]: Advancing to term 9
I20251028 14:52:13.486724 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 8.
I20251028 14:52:13.488147 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 9.
I20251028 14:52:13.488745 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.488832 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.488946 3704 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.489048 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 8 FOLLOWER]: Advancing to term 9
I20251028 14:52:13.489140 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 9 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20251028 14:52:13.489183 3781 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 FOLLOWER]: Leader election won for term 8
I20251028 14:52:13.489382 3781 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.489413 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.489697 3781 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.489835 3799 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 FOLLOWER]: Leader election won for term 9
I20251028 14:52:13.490046 3799 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.490319 3799 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 6.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.491865 3704 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.492486 3704 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 9 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.492955 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 9 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.492787 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 7 to 8, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 8 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.493062 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 9 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.493136 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 8 FOLLOWER]: Advancing to term 9
I20251028 14:52:13.493227 3564 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 LEADER]: Stepping down as leader of term 8
I20251028 14:52:13.493358 3564 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.493654 3564 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.493875 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.494081 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.494232 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 9 FOLLOWER]: Advancing to term 10
I20251028 14:52:13.494797 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 8 FOLLOWER]: Advancing to term 9
I20251028 14:52:13.495708 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 9 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.497434 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.498193 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 8 to 9, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 9 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.498500 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 9 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.498899 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 10 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.499080 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 9 FOLLOWER]: Advancing to term 10
I20251028 14:52:13.499184 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.499496 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20251028 14:52:13.500152 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 10 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.500543 3564 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 LEADER]: Stepping down as leader of term 9
I20251028 14:52:13.500664 3564 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.500994 3564 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.501310 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 FOLLOWER]: Advancing to term 10
I20251028 14:52:13.501901 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 10 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.502206 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 10 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.503763 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 10 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.503859 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.504261 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.504685 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20251028 14:52:13.504938 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.505111 3427 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.505211 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 9 FOLLOWER]: Advancing to term 10
I20251028 14:52:13.507375 3427 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.508001 3427 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 10 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.508566 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.508778 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 9 FOLLOWER]: Advancing to term 10
I20251028 14:52:13.509135 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.509372 3427 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.509474 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 FOLLOWER]: Advancing to term 11
I20251028 14:52:13.510910 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 10 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20251028 14:52:13.511346 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.511564 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 9 FOLLOWER]: Advancing to term 10
I20251028 14:52:13.511475 3427 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.512405 3427 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 11 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.512957 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.513154 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 10 FOLLOWER]: Advancing to term 11
I20251028 14:52:13.513106 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 11 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.513443 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 10 FOLLOWER]: Advancing to term 11
I20251028 14:52:13.513504 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.514201 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 10.
I20251028 14:52:13.514789 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.515084 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.515208 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 10 FOLLOWER]: Advancing to term 11
I20251028 14:52:13.516008 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 11.
I20251028 14:52:13.516664 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.516968 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 11 because replica has last-logged OpId of term: 9 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 3.
I20251028 14:52:13.517482 3785 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 FOLLOWER]: Leader election won for term 10
I20251028 14:52:13.517534 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.517783 3785 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.517992 3774 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 FOLLOWER]: Leader election won for term 11
I20251028 14:52:13.518131 3785 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 7.3, Last appended by leader: 3, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.518316 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.518988 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 11 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.518138 3774 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.519686 3774 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.521587 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 11 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.521741 3427 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 LEADER]: Stepping down as leader of term 10
I20251028 14:52:13.521711 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 11 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.521826 3427 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.521934 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 10 FOLLOWER]: Advancing to term 11
I20251028 14:52:13.521700 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 8 to 10, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 10 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.522042 3427 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 10.4, Last appended by leader: 4, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.522266 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 10 FOLLOWER]: Advancing to term 11
I20251028 14:52:13.523175 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.523388 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.523510 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 FOLLOWER]: Advancing to term 12
I20251028 14:52:13.525358 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 11.
I20251028 14:52:13.526050 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.526413 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 FOLLOWER]: Leader election won for term 11
I20251028 14:52:13.526592 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.526865 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.526962 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.527596 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 12 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.528236 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 11 because replica has last-logged OpId of term: 10 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 3.
I20251028 14:52:13.528594 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 9 to 11, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 11 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.529419 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.529804 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.529775 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 10 to 11, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 11 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.530000 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 LEADER]: Stepping down as leader of term 11
I20251028 14:52:13.530104 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.530391 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.530378 3703 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.530683 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 FOLLOWER]: Advancing to term 12
I20251028 14:52:13.530719 3703 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.530997 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 11 FOLLOWER]: Advancing to term 12
I20251028 14:52:13.532965 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 12 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
I20251028 14:52:13.533987 3703 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.535375 3703 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 12 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.535437 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 12 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.535614 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 11 FOLLOWER]: Advancing to term 12
I20251028 14:52:13.530719 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.535822 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 11 FOLLOWER]: Advancing to term 12
I20251028 14:52:13.535593 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 12 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.536190 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 LEADER]: Stepping down as leader of term 11
I20251028 14:52:13.536283 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.536453 3703 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.536609 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.536921 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 11 FOLLOWER]: Advancing to term 12
I20251028 14:52:13.538874 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 12 because replica has last-logged OpId of term: 10 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.539047 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 12.
I20251028 14:52:13.539253 3703 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.539374 3703 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 12 FOLLOWER]: Advancing to term 13
I20251028 14:52:13.539618 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 12 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.539592 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.540057 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.540205 3781 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 FOLLOWER]: Leader election won for term 12
I20251028 14:52:13.540344 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20251028 14:52:13.540369 3781 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.540652 3781 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 9.4, Last appended by leader: 4, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.542161 3703 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.542857 3703 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 13 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.543375 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 13 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.543610 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 12 FOLLOWER]: Advancing to term 13
I20251028 14:52:13.543547 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 11 to 12, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 12 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.544597 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.544898 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 13 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.545096 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 LEADER]: Stepping down as leader of term 12
I20251028 14:52:13.545172 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.545403 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.545598 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.545540 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 12.5, Last appended by leader: 5, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.545730 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 12 FOLLOWER]: Advancing to term 13
I20251028 14:52:13.545799 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 FOLLOWER]: Advancing to term 13
I20251028 14:52:13.548055 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.548910 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 13 because replica has last-logged OpId of term: 12 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.549082 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 13 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.549156 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.549360 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 12 FOLLOWER]: Advancing to term 13
I20251028 14:52:13.550263 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.550848 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 13 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.551018 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.551138 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 FOLLOWER]: Advancing to term 14
I20251028 14:52:13.551729 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 13 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 4.
I20251028 14:52:13.552254 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 13 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.552429 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 12 FOLLOWER]: Advancing to term 13
I20251028 14:52:13.553094 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.553716 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 14 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.554319 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.554522 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 13 FOLLOWER]: Advancing to term 14
I20251028 14:52:13.554708 3703 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 13.
I20251028 14:52:13.555054 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.555156 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.555473 3774 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 FOLLOWER]: Leader election won for term 13
I20251028 14:52:13.555665 3774 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.556017 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.556365 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.556169 3774 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 10.4, Last appended by leader: 4, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.557209 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.557525 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.557696 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 13 FOLLOWER]: Advancing to term 14
I20251028 14:52:13.558125 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 14.
I20251028 14:52:13.558307 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20251028 14:52:13.559265 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.559653 3774 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 FOLLOWER]: Leader election won for term 14
I20251028 14:52:13.559800 3774 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.560093 3774 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.560179 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.560137 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 11 to 13, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 13 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.560876 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 14 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.561447 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.561642 3428 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 LEADER]: Stepping down as leader of term 13
I20251028 14:52:13.561753 3428 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.561793 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.562005 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 13 FOLLOWER]: Advancing to term 14
I20251028 14:52:13.562048 3428 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 13.5, Last appended by leader: 5, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.562358 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 13 FOLLOWER]: Advancing to term 14
I20251028 14:52:13.562563 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.562742 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.562868 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 13 FOLLOWER]: Advancing to term 14
I20251028 14:52:13.564831 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.564874 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 14.
I20251028 14:52:13.565320 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.565611 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 14 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.565672 3781 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 FOLLOWER]: Leader election won for term 14
I20251028 14:52:13.565846 3781 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.566103 3781 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.566145 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.566439 3704 raft_consensus.cc:2393] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in current term 14: Already voted for candidate d7cc28c72b6e454f98cc0096e3c57a1d in this term.
I20251028 14:52:13.566586 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 12 to 14, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 14 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.567907 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 14 because replica has last-logged OpId of term: 13 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 4.
I20251028 14:52:13.568876 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.569142 3428 raft_consensus.cc:2393] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 LEADER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in current term 14: Already voted for candidate d7cc28c72b6e454f98cc0096e3c57a1d in this term.
I20251028 14:52:13.569849 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.570294 3799 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20251028 14:52:13.570465 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 13 to 14, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 14 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.571646 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.572506 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.572768 3704 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.572877 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Advancing to term 15
I20251028 14:52:13.575021 3704 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.575623 3704 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 15 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.576480 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 15 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.576498 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 15 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.576697 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 FOLLOWER]: Advancing to term 15
I20251028 14:52:13.576681 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.576865 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 LEADER]: Stepping down as leader of term 14
I20251028 14:52:13.576905 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.577045 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 14 FOLLOWER]: Advancing to term 15
I20251028 14:52:13.577036 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.577400 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.577728 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 FOLLOWER]: Advancing to term 15
I20251028 14:52:13.580070 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.580199 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 15 because replica has last-logged OpId of term: 13 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.580682 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 15 because replica has last-logged OpId of term: 14 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.581100 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.581354 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 15 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.581357 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 15 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.581459 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20251028 14:52:13.581552 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 14 FOLLOWER]: Advancing to term 15
I20251028 14:52:13.581549 3428 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 LEADER]: Stepping down as leader of term 14
I20251028 14:52:13.581610 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 15 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.581632 3428 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.581956 3428 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.582284 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 14 FOLLOWER]: Advancing to term 15
I20251028 14:52:13.583259 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.583901 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 15 because replica has last-logged OpId of term: 12 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.584183 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.584343 3427 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.584434 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 15 FOLLOWER]: Advancing to term 16
I20251028 14:52:13.584486 3428 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 15 because replica has last-logged OpId of term: 14 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.584877 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.585157 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20251028 14:52:13.586481 3427 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.587059 3427 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 16 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.587582 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 5 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.587667 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 16 candidate_status { last_received { term: 13 index: 5 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.587807 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 15 FOLLOWER]: Advancing to term 16
I20251028 14:52:13.587857 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 15 FOLLOWER]: Advancing to term 16
I20251028 14:52:13.588050 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.588227 3427 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.588325 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 15 FOLLOWER]: Advancing to term 16
I20251028 14:52:13.590284 3427 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.590803 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 16.
I20251028 14:52:13.590883 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 16 because replica has last-logged OpId of term: 14 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 5.
I20251028 14:52:13.591395 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.591413 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 16 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.591493 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.591574 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 15 FOLLOWER]: Advancing to term 16
I20251028 14:52:13.591583 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 15 FOLLOWER]: Advancing to term 16
I20251028 14:52:13.591884 3774 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 FOLLOWER]: Leader election won for term 16
I20251028 14:52:13.592016 3774 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.592240 3774 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 13.5, Last appended by leader: 5, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.593662 3427 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 16 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.594200 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.594520 3563 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 16.
I20251028 14:52:13.594542 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 16.
I20251028 14:52:13.594818 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 14 to 16, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 16 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.595021 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.595180 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.595155 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.595265 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 16 FOLLOWER]: Advancing to term 17
I20251028 14:52:13.595433 3785 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 FOLLOWER]: Leader election won for term 16
I20251028 14:52:13.595592 3785 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.595831 3785 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.597741 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.598327 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 17 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.598995 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.599027 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.599193 3427 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 LEADER]: Stepping down as leader of term 16
I20251028 14:52:13.599298 3427 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.599295 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 16 FOLLOWER]: Advancing to term 17
I20251028 14:52:13.599578 3427 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 16.6, Last appended by leader: 6, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.599875 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 FOLLOWER]: Advancing to term 17
I20251028 14:52:13.599903 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 14 to 16. New cstate: current_term: 16 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.602110 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 17.
I20251028 14:52:13.602537 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.602623 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 17 because replica has last-logged OpId of term: 16 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 5.
I20251028 14:52:13.602923 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 FOLLOWER]: Leader election won for term 17
I20251028 14:52:13.603093 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.603345 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.604172 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.604372 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.604501 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 16 FOLLOWER]: Advancing to term 17
I20251028 14:52:13.606055 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 16 to 17, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 17 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.607184 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.607702 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 17 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.608670 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 17 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.608686 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 17 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.608880 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 LEADER]: Stepping down as leader of term 16
I20251028 14:52:13.608992 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.608997 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 16 FOLLOWER]: Advancing to term 17
I20251028 14:52:13.609274 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 16.6, Last appended by leader: 6, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.609723 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 16 FOLLOWER]: Advancing to term 17
I20251028 14:52:13.611318 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.611413 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 17.
I20251028 14:52:13.611857 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 17 because replica has last-logged OpId of term: 16 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 5.
I20251028 14:52:13.611897 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.612208 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.612399 3704 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.612489 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 17 FOLLOWER]: Advancing to term 18
I20251028 14:52:13.612548 3799 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 FOLLOWER]: Leader election won for term 17
I20251028 14:52:13.612699 3799 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.612983 3799 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 12.5, Last appended by leader: 5, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.615048 3704 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.615844 3704 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 18 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.615712 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 16 to 17, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 17 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.616434 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 18 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.616453 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 18 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.616653 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 LEADER]: Stepping down as leader of term 17
I20251028 14:52:13.616743 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.616969 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.617058 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.617152 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.617312 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 17 FOLLOWER]: Advancing to term 18
I20251028 14:52:13.617348 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 FOLLOWER]: Advancing to term 18
I20251028 14:52:13.616638 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 17 FOLLOWER]: Advancing to term 18
I20251028 14:52:13.620384 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 18 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.620842 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 18 because replica has last-logged OpId of term: 16 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.620766 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.621295 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.621620 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 18 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.621652 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20251028 14:52:13.622095 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 18 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.622200 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 18 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.622256 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 LEADER]: Stepping down as leader of term 17
I20251028 14:52:13.622349 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.622352 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.622398 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 17 FOLLOWER]: Advancing to term 18
I20251028 14:52:13.622632 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.622929 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 17 FOLLOWER]: Advancing to term 18
I20251028 14:52:13.623108 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.623298 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.623402 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 18 FOLLOWER]: Advancing to term 19
I20251028 14:52:13.625002 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 18 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.625399 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 18 because replica has last-logged OpId of term: 16 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.625839 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.626159 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20251028 14:52:13.626211 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.626793 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 19 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.627209 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 6 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.627277 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 6 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.627382 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 18 FOLLOWER]: Advancing to term 19
I20251028 14:52:13.627462 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 18 FOLLOWER]: Advancing to term 19
I20251028 14:52:13.627890 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.628062 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.628175 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 18 FOLLOWER]: Advancing to term 19
I20251028 14:52:13.629582 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 19 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 6.
I20251028 14:52:13.630316 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 19.
I20251028 14:52:13.630810 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.630978 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.631142 3785 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 FOLLOWER]: Leader election won for term 19
I20251028 14:52:13.631289 3785 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.631603 3785 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 16.6, Last appended by leader: 6, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.631966 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 19 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.632256 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 6 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.632445 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 18 FOLLOWER]: Advancing to term 19
I20251028 14:52:13.633319 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 19 candidate_status { last_received { term: 16 index: 6 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.633514 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 18 FOLLOWER]: Advancing to term 19
I20251028 14:52:13.633947 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 17 to 19, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 19 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.635248 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 19 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 6.
I20251028 14:52:13.636719 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.637293 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 19.
I20251028 14:52:13.637728 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.637799 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.637995 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.638136 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 19 FOLLOWER]: Advancing to term 20
I20251028 14:52:13.638091 3785 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 FOLLOWER]: Leader election won for term 19
I20251028 14:52:13.638286 3785 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.638602 3785 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 16.6, Last appended by leader: 6, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.641471 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 17 to 19, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 19 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.641546 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.642287 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 20 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.642870 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.642900 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.643074 3428 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 LEADER]: Stepping down as leader of term 19
I20251028 14:52:13.643106 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 19 FOLLOWER]: Advancing to term 20
I20251028 14:52:13.643157 3428 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.643473 3428 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 19.7, Last appended by leader: 7, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.643767 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 FOLLOWER]: Advancing to term 20
I20251028 14:52:13.645890 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 20.
I20251028 14:52:13.646307 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 20 because replica has last-logged OpId of term: 19 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 6.
I20251028 14:52:13.646344 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.646695 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 FOLLOWER]: Leader election won for term 20
I20251028 14:52:13.646692 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.646889 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.646973 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.647087 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 19 FOLLOWER]: Advancing to term 20
I20251028 14:52:13.647176 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.649861 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.650044 3785 raft_consensus.cc:493] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:13.650172 3785 raft_consensus.cc:3060] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.650197 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 19 to 20, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 20 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.650959 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 20 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.651082 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.651335 3428 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 LEADER]: Stepping down as leader of term 19
I20251028 14:52:13.651448 3428 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.651866 3428 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 19.7, Last appended by leader: 7, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.652093 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 19 FOLLOWER]: Advancing to term 20
I20251028 14:52:13.653116 3785 raft_consensus.cc:515] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.654393 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.654445 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ee80079eb8c42578768d1d87767ed19" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.654397 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5ee80079eb8c42578768d1d87767ed19" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.654642 3704 raft_consensus.cc:3060] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.654661 3563 raft_consensus.cc:3060] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:13.654933 3428 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 20 because replica has last-logged OpId of term: 19 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 6.
I20251028 14:52:13.654656 3703 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 19 FOLLOWER]: Advancing to term 20
I20251028 14:52:13.657166 3704 raft_consensus.cc:2468] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 1.
I20251028 14:52:13.657891 3563 raft_consensus.cc:2468] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 1.
I20251028 14:52:13.658422 3703 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 20.
I20251028 14:52:13.658493 3353 leader_election.cc:304] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.658921 3774 raft_consensus.cc:2804] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:13.658941 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.658890 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.659068 3774 raft_consensus.cc:697] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [term 1 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.659235 3781 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 FOLLOWER]: Leader election won for term 20
I20251028 14:52:13.659394 3781 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.659399 3774 consensus_queue.cc:237] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.659669 3781 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.660358 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.660575 3704 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.660679 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 20 FOLLOWER]: Advancing to term 21
I20251028 14:52:13.661866 3809 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: TxnStatusTablet state changed. Reason: New leader d7cc28c72b6e454f98cc0096e3c57a1d. Latest consensus state: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.662189 3809 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:13.662950 3811 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:13.662956 3785 leader_election.cc:290] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 1 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.663445 3811 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:13.663888 3704 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.664286 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 19 to 20, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 20 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.664383 3263 catalog_manager.cc:5649] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.664906 3704 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 21 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.666057 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 21 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.666282 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 LEADER]: Stepping down as leader of term 20
I20251028 14:52:13.666252 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 21 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.666355 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.666445 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 20 FOLLOWER]: Advancing to term 21
I20251028 14:52:13.667801 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.668179 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 FOLLOWER]: Advancing to term 21
I20251028 14:52:13.670264 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 21 because replica has last-logged OpId of term: 19 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.672188 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 21 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.672485 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.672703 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.672832 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 20 FOLLOWER]: Advancing to term 21
I20251028 14:52:13.674953 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.675637 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 21 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.676306 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.676271 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 21 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.676506 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 20 FOLLOWER]: Advancing to term 21
I20251028 14:52:13.676607 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20251028 14:52:13.677007 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 21 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.677215 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 LEADER]: Stepping down as leader of term 20
I20251028 14:52:13.677304 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.677613 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.677914 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 20 FOLLOWER]: Advancing to term 21
I20251028 14:52:13.679020 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.680730 3428 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 21 because replica has last-logged OpId of term: 19 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.680859 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 21 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.681162 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.681463 3427 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.681463 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.681612 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 21 FOLLOWER]: Advancing to term 22
I20251028 14:52:13.681818 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20251028 14:52:13.683902 3427 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.684667 3427 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 22 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.685178 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.685323 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.685549 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 21 FOLLOWER]: Advancing to term 22
I20251028 14:52:13.685658 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 21 FOLLOWER]: Advancing to term 22
I20251028 14:52:13.685777 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.686375 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.686538 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 21 FOLLOWER]: Advancing to term 22
I20251028 14:52:13.688743 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 22 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 7.
I20251028 14:52:13.688764 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 22.
I20251028 14:52:13.689395 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.689540 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.689745 3809 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 FOLLOWER]: Leader election won for term 22
I20251028 14:52:13.689913 3809 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.690189 3809 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 19.7, Last appended by leader: 7, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.690654 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.690701 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.690852 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 21 FOLLOWER]: Advancing to term 22
I20251028 14:52:13.690853 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 21 FOLLOWER]: Advancing to term 22
I20251028 14:52:13.692734 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 20 to 22, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 22 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.693710 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 22.
I20251028 14:52:13.693827 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 22 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 7.
I20251028 14:52:13.694160 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.694482 3809 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 FOLLOWER]: Leader election won for term 22
I20251028 14:52:13.694662 3809 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.694872 3809 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 19.7, Last appended by leader: 7, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.694474 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 22 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.698483 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 20 to 22, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 22 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.699409 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.700376 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.700639 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.700748 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 22 FOLLOWER]: Advancing to term 23
I20251028 14:52:13.703418 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.704169 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 23 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.705052 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.705070 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.705361 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 22 FOLLOWER]: Advancing to term 23
I20251028 14:52:13.705348 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.705519 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.705492 3428 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 LEADER]: Stepping down as leader of term 22
I20251028 14:52:13.705641 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 22 FOLLOWER]: Advancing to term 23
I20251028 14:52:13.705808 3428 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.706168 3428 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 22.8, Last appended by leader: 8, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.706532 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 FOLLOWER]: Advancing to term 23
I20251028 14:52:13.708461 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 23.
I20251028 14:52:13.708901 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.709045 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:13.709408 3781 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 FOLLOWER]: Leader election won for term 23
I20251028 14:52:13.709586 3781 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.709894 3781 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.710029 3428 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 23 because replica has last-logged OpId of term: 22 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 7.
I20251028 14:52:13.710629 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.710983 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 LEADER]: Stepping down as leader of term 22
I20251028 14:52:13.711076 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.711395 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 22.8, Last appended by leader: 8, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.711808 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 22 FOLLOWER]: Advancing to term 23
I20251028 14:52:13.713475 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 22 to 23, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 23 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.715495 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 23 because replica has last-logged OpId of term: 22 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 7.
I20251028 14:52:13.717087 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.717335 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 22 FOLLOWER]: Advancing to term 23
I20251028 14:52:13.717412 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 23 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.719931 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 23.
I20251028 14:52:13.720454 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.720609 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.721021 3799 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 FOLLOWER]: Leader election won for term 23
I20251028 14:52:13.721182 3799 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.721556 3799 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.721901 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.722193 3704 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.722306 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 23 FOLLOWER]: Advancing to term 24
I20251028 14:52:13.724355 3704 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.725070 3704 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 24 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.725785 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 24 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.726017 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 23 FOLLOWER]: Advancing to term 24
I20251028 14:52:13.726528 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 24 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.726825 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 LEADER]: Stepping down as leader of term 23
I20251028 14:52:13.726945 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.727295 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.727662 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 FOLLOWER]: Advancing to term 24
I20251028 14:52:13.727656 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 22 to 23, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 23 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.729486 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 24 because replica has last-logged OpId of term: 22 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.730335 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.730558 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.730700 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 23 FOLLOWER]: Advancing to term 24
I20251028 14:52:13.732151 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 24 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.732988 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.733343 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20251028 14:52:13.733561 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.734742 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 24 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.734977 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 23 FOLLOWER]: Advancing to term 24
I20251028 14:52:13.735169 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 24 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.735603 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 24 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.735821 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 LEADER]: Stepping down as leader of term 23
I20251028 14:52:13.735913 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.736203 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.736590 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 23 FOLLOWER]: Advancing to term 24
I20251028 14:52:13.738581 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 24 because replica has last-logged OpId of term: 22 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.740391 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 24 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.740769 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.741025 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.741425 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20251028 14:52:13.742208 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.742470 3427 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.742616 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 24 FOLLOWER]: Advancing to term 25
I20251028 14:52:13.745874 3427 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.747107 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 8 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.747107 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 8 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.747352 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 24 FOLLOWER]: Advancing to term 25
I20251028 14:52:13.747360 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 24 FOLLOWER]: Advancing to term 25
I20251028 14:52:13.750538 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 25.
I20251028 14:52:13.750629 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 25 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 8.
I20251028 14:52:13.751120 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:13.751492 3785 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 FOLLOWER]: Leader election won for term 25
I20251028 14:52:13.751667 3785 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.752019 3785 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 22.8, Last appended by leader: 8, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.753571 3427 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 25 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.755383 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.755589 3427 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.755740 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 24 FOLLOWER]: Advancing to term 25
I20251028 14:52:13.755908 3264 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 23 to 25, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 25 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.758586 3427 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.759935 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 8 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.760150 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 24 FOLLOWER]: Advancing to term 25
I20251028 14:52:13.761585 3704 raft_consensus.cc:1275] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c [term 1 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:13.762480 3809 consensus_queue.cc:1048] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:13.762995 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 25 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 8.
I20251028 14:52:13.764330 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 25 candidate_status { last_received { term: 22 index: 8 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.764393 3563 raft_consensus.cc:1275] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 1 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:13.764531 3703 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 24 FOLLOWER]: Advancing to term 25
I20251028 14:52:13.767772 3703 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 25.
I20251028 14:52:13.768510 3785 consensus_queue.cc:1048] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:13.769232 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.769656 3785 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 FOLLOWER]: Leader election won for term 25
I20251028 14:52:13.769812 3785 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.770129 3785 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 22.8, Last appended by leader: 8, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.774830 3264 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 23 to 25, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 25 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.780126 3427 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 25 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.783557 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.784749 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.785084 3564 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.785239 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 25 FOLLOWER]: Advancing to term 26
I20251028 14:52:13.788854 3799 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: TxnStatusTablet state changed. Reason: New leader d7cc28c72b6e454f98cc0096e3c57a1d. Latest consensus state: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.789141 3799 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:13.791468 3776 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: TxnStatusTablet state changed. Reason: New leader d7cc28c72b6e454f98cc0096e3c57a1d. Latest consensus state: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.791756 3776 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:13.792369 3814 mvcc.cc:204] Tried to move back new op lower bound from 7215772195876442112 to 7215772195471503360. Current Snapshot: MvccSnapshot[applied={T|T < 7215772195876442112}]
I20251028 14:52:13.792765 3564 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.794127 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.794431 3427 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 LEADER]: Stepping down as leader of term 25
I20251028 14:52:13.794550 3427 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.794905 3427 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 25.9, Last appended by leader: 9, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.795226 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 FOLLOWER]: Advancing to term 26
I20251028 14:52:13.799093 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 26 because replica has last-logged OpId of term: 25 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 8.
I20251028 14:52:13.801052 3564 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 26 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.801939 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.802189 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 25 FOLLOWER]: Advancing to term 26
I20251028 14:52:13.802806 3817 mvcc.cc:204] Tried to move back new op lower bound from 7215772195876442112 to 7215772195471503360. Current Snapshot: MvccSnapshot[applied={T|T < 7215772195876442112}]
I20251028 14:52:13.805146 3703 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 26.
I20251028 14:52:13.805755 3774 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.806030 3774 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:13.803897 3785 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.807166 3785 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P d7cc28c72b6e454f98cc0096e3c57a1d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:13.808203 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.808698 3781 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 FOLLOWER]: Leader election won for term 26
I20251028 14:52:13.808853 3781 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.809135 3781 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.814615 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.814873 3564 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.814987 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 25 FOLLOWER]: Advancing to term 26
I20251028 14:52:13.818377 3564 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.819794 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.820058 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 LEADER]: Stepping down as leader of term 25
I20251028 14:52:13.820248 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.820631 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 25.9, Last appended by leader: 9, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.821012 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 25 FOLLOWER]: Advancing to term 26
I20251028 14:52:13.824769 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 26 because replica has last-logged OpId of term: 25 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 8.
I20251028 14:52:13.826512 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.826797 3703 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 25 FOLLOWER]: Advancing to term 26
I20251028 14:52:13.829989 3703 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 26.
I20251028 14:52:13.830788 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.831223 3773 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 FOLLOWER]: Leader election won for term 26
I20251028 14:52:13.831362 3773 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.831715 3773 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.833350 3564 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 26 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.837221 3799 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.837462 3799 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:13.840188 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.843636 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 25 to 26, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 26 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.844210 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 25 to 26, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 26 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.845404 3703 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.845665 3703 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.845780 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 26 FOLLOWER]: Advancing to term 27
I20251028 14:52:13.855661 3703 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.857299 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.857676 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 26 FOLLOWER]: Advancing to term 27
I20251028 14:52:13.861209 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 27 because replica has last-logged OpId of term: 25 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.867208 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.867475 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 LEADER]: Stepping down as leader of term 26
I20251028 14:52:13.867591 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.867914 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.867905 3776 tablet_replica.cc:442] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
I20251028 14:52:13.868105 3776 tablet_replica.cc:445] T 5ee80079eb8c42578768d1d87767ed19 P d400e7c613e146ab91729d201413b76c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:13.868209 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 FOLLOWER]: Advancing to term 27
I20251028 14:52:13.871881 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 27 because replica has last-logged OpId of term: 26 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.872275 3703 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 27 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.873378 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.874105 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20251028 14:52:13.875151 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.875381 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.875512 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 26 FOLLOWER]: Advancing to term 27
I20251028 14:52:13.879065 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.880285 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 27 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:13.881402 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.881732 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 26 FOLLOWER]: Advancing to term 27
I20251028 14:52:13.885226 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 27 because replica has last-logged OpId of term: 25 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.886049 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.886274 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 LEADER]: Stepping down as leader of term 26
I20251028 14:52:13.886381 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.886701 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.887018 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 26 FOLLOWER]: Advancing to term 27
I20251028 14:52:13.888314 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:13.889357 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.889593 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 27 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.889755 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 27 FOLLOWER]: Advancing to term 28
I20251028 14:52:13.890895 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 27 because replica has last-logged OpId of term: 26 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:13.892881 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.893702 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.894199 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20251028 14:52:13.895361 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 28 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.895978 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.896271 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 27 FOLLOWER]: Advancing to term 28
I20251028 14:52:13.896621 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:13.896854 3427 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 27 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.896822 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.897006 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 27 FOLLOWER]: Advancing to term 28
I20251028 14:52:13.896991 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 27 FOLLOWER]: Advancing to term 28
I20251028 14:52:13.900256 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 28 because replica has last-logged OpId of term: 26 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 9.
I20251028 14:52:13.900785 3427 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.902031 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:13.902238 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 27 FOLLOWER]: Advancing to term 28
I20251028 14:52:13.905874 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 28 because replica has last-logged OpId of term: 26 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 9.
I20251028 14:52:13.906487 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 28.
I20251028 14:52:13.908425 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.908924 3785 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 FOLLOWER]: Leader election won for term 28
I20251028 14:52:13.909154 3785 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.909473 3785 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 25.9, Last appended by leader: 9, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.912880 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 26 to 28, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 28 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.923532 3427 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 28 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.929356 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:13.934154 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.935006 3703 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 27 FOLLOWER]: Advancing to term 28
I20251028 14:52:13.938408 3703 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 28.
I20251028 14:52:13.945585 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:13.945585 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.945869 3564 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.945982 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 28 FOLLOWER]: Advancing to term 29
I20251028 14:52:13.945992 3809 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 FOLLOWER]: Leader election won for term 28
I20251028 14:52:13.946166 3809 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.946461 3809 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 25.9, Last appended by leader: 9, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.949295 3564 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.950165 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 26 to 28, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 28 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.950515 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 29 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.950752 3427 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 LEADER]: Stepping down as leader of term 28
I20251028 14:52:13.950862 3427 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.951152 3427 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.951478 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 FOLLOWER]: Advancing to term 29
I20251028 14:52:13.955010 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 29 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 9.
I20251028 14:52:13.969770 3564 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 29 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.969864 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 29 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.970090 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 28 FOLLOWER]: Advancing to term 29
I20251028 14:52:13.970999 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:13.971239 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.971359 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 28 FOLLOWER]: Advancing to term 29
I20251028 14:52:13.973568 3703 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 29.
I20251028 14:52:13.974256 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.974454 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.974794 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 FOLLOWER]: Leader election won for term 29
I20251028 14:52:13.974958 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.975257 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.975705 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 29 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:13.975908 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 LEADER]: Stepping down as leader of term 28
I20251028 14:52:13.976006 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:13.976318 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.976604 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 28 FOLLOWER]: Advancing to term 29
I20251028 14:52:13.979928 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 29 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 9.
I20251028 14:52:13.979699 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 28 to 29, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 29 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.981230 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 29 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:13.983428 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 29 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:13.983673 3703 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 28 FOLLOWER]: Advancing to term 29
I20251028 14:52:13.986614 3703 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 29.
I20251028 14:52:13.987251 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:13.987612 3781 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 FOLLOWER]: Leader election won for term 29
I20251028 14:52:13.987799 3781 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:13.988116 3781 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:13.995016 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:13.996053 3703 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:13.996307 3703 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:13.996433 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 29 FOLLOWER]: Advancing to term 30
I20251028 14:52:13.996205 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 28 to 29, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 29 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:13.999399 3703 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.000727 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.001016 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 29 FOLLOWER]: Advancing to term 30
I20251028 14:52:14.005713 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 30 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:14.008582 3703 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 30 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.009729 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.009958 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 LEADER]: Stepping down as leader of term 29
I20251028 14:52:14.010066 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.010246 3703 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.010365 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 29.10, Last appended by leader: 10, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.010531 3703 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.010707 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 FOLLOWER]: Advancing to term 30
I20251028 14:52:14.010815 3703 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 29 FOLLOWER]: Advancing to term 30
I20251028 14:52:14.014458 3703 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.015714 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.015946 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 29 FOLLOWER]: Advancing to term 30
W20251028 14:52:14.017448 3830 tablet_replica.cc:1306] Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER: cancelling pending write operations
I20251028 14:52:14.017531 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 30 because replica has last-logged OpId of term: 29 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20251028 14:52:14.017736 3830 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.018299 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.018774 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20251028 14:52:14.019133 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 30 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:14.020162 3703 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 30 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.020820 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.021039 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 LEADER]: Stepping down as leader of term 29
I20251028 14:52:14.021163 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.021543 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 29.10, Last appended by leader: 10, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.021831 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 29 FOLLOWER]: Advancing to term 30
I20251028 14:52:14.024974 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 30 because replica has last-logged OpId of term: 29 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:14.025720 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.026211 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20251028 14:52:14.027082 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:14.027920 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.028139 3427 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 30 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.028277 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 30 FOLLOWER]: Advancing to term 31
W20251028 14:52:14.033015 3831 tablet_replica.cc:1306] Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER: cancelling pending write operations
W20251028 14:52:14.033329 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.036144 3835 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.039958 3427 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 31 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:14.041576 3833 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.046103 3830 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.049088 3832 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.050644 3834 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.051455 3830 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.054863 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.059343 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.061998 3427 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 31 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.062446 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.062832 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 30 FOLLOWER]: Advancing to term 31
I20251028 14:52:14.063632 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.063879 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 30 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.063992 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 30 FOLLOWER]: Advancing to term 31
I20251028 14:52:14.066243 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 31 because replica has last-logged OpId of term: 29 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 10.
I20251028 14:52:14.067095 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 31 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.068240 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.068493 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 30 FOLLOWER]: Advancing to term 31
I20251028 14:52:14.069447 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 31 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.072338 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 31 because replica has last-logged OpId of term: 29 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 10.
I20251028 14:52:14.076242 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:14.078415 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.078709 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.078861 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 31 FOLLOWER]: Advancing to term 32
I20251028 14:52:14.082851 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.083389 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.083611 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 30 FOLLOWER]: Advancing to term 31
I20251028 14:52:14.084007 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.084192 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 30 FOLLOWER]: Advancing to term 31
I20251028 14:52:14.087445 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 32 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.088826 3702 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 32 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.089154 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 32 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.089422 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 31 FOLLOWER]: Advancing to term 32
I20251028 14:52:14.089933 3703 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 31.
I20251028 14:52:14.092729 3427 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 32.
I20251028 14:52:14.090240 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.093581 3564 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.093730 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 31 FOLLOWER]: Advancing to term 32
I20251028 14:52:14.093734 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 31.
I20251028 14:52:14.095363 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:14.096352 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 FOLLOWER]: Leader election won for term 32
I20251028 14:52:14.096555 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.096915 3564 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.097061 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 29.10, Last appended by leader: 10, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.097671 3564 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 32 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.100399 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 29 to 32. New cstate: current_term: 32 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.102672 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 32 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.102923 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 31 FOLLOWER]: Advancing to term 32
I20251028 14:52:14.106528 3427 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 32.
I20251028 14:52:14.107451 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 32 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.107771 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 31 FOLLOWER]: Advancing to term 32
I20251028 14:52:14.108424 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:14.109715 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.109984 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.110117 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 31 FOLLOWER]: Advancing to term 32
I20251028 14:52:14.111238 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 32.
I20251028 14:52:14.111786 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:14.112458 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:14.112936 3809 raft_consensus.cc:2764] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Leader election decision vote started in defunct term 31: won
I20251028 14:52:14.113595 3809 raft_consensus.cc:2764] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Leader election decision vote started in defunct term 31: won
I20251028 14:52:14.114164 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:14.114601 3781 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 FOLLOWER]: Leader election won for term 32
I20251028 14:52:14.114800 3781 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
W20251028 14:52:14.114979 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.115149 3781 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 29.10, Last appended by leader: 10, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.115662 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.116575 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 32 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.118152 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 32 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.118464 3428 raft_consensus.cc:2393] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c in current term 32: Already voted for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in this term.
I20251028 14:52:14.121811 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 29 to 32. New cstate: current_term: 32 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.123725 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 32 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.124089 3564 raft_consensus.cc:2393] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 LEADER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c in current term 32: Already voted for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in this term.
I20251028 14:52:14.124624 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.125185 3776 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20251028 14:52:14.127779 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.130859 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 32 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.131037 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 32 FOLLOWER]: Advancing to term 33
I20251028 14:52:14.138471 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 33 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.139312 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 33 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.140268 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 33 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.140548 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Advancing to term 33
I20251028 14:52:14.142874 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 33 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.143173 3564 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 LEADER]: Stepping down as leader of term 32
I20251028 14:52:14.143255 3564 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.143579 3564 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 32.11, Last appended by leader: 11, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.143944 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 FOLLOWER]: Advancing to term 33
I20251028 14:52:14.147028 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 33 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:14.147567 3564 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 33 because replica has last-logged OpId of term: 32 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251028 14:52:14.148279 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.148721 3776 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20251028 14:52:14.158370 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:14.159471 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.159792 3427 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.159946 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 32 FOLLOWER]: Advancing to term 33
I20251028 14:52:14.162667 3427 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.164002 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 33 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.164026 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 33 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.164258 3564 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 LEADER]: Stepping down as leader of term 32
I20251028 14:52:14.164321 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 32 FOLLOWER]: Advancing to term 33
I20251028 14:52:14.164356 3564 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.164662 3564 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 32.11, Last appended by leader: 11, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.164950 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 32 FOLLOWER]: Advancing to term 33
I20251028 14:52:14.167426 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 33.
I20251028 14:52:14.168017 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 33 because replica has last-logged OpId of term: 32 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 10.
I20251028 14:52:14.167940 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:14.168378 3785 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 FOLLOWER]: Leader election won for term 33
I20251028 14:52:14.168529 3785 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.168831 3785 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.171345 3427 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 33 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.172421 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 32 to 33, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 33 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.172811 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.173238 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.173461 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 FOLLOWER]: Advancing to term 34
I20251028 14:52:14.179157 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.180667 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 34 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.180944 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 33 FOLLOWER]: Advancing to term 34
I20251028 14:52:14.183682 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 34 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.184136 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 34 because replica has last-logged OpId of term: 32 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 10.
I20251028 14:52:14.185000 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 34 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.185231 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 33 FOLLOWER]: Advancing to term 34
W20251028 14:52:14.186616 3851 tablet_replica.cc:1306] Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER: cancelling pending write operations
W20251028 14:52:14.186836 3851 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.187557 3845 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.190886 3849 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.191740 3853 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.193290 3847 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.200481 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
W20251028 14:52:14.204064 3846 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.206609 3842 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.206668 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.208876 3704 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 33 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 33 index: 12. (index mismatch)
W20251028 14:52:14.209205 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.210862 3774 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:14.210330 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 34.
I20251028 14:52:14.212270 3564 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 33 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 32 index: 11. Preceding OpId from leader: term: 33 index: 12. (index mismatch)
I20251028 14:52:14.212815 3785 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 0, Time since last communication: 0.000s
W20251028 14:52:14.212954 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.213038 3785 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 32.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251028 14:52:14.213421 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.213711 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.213833 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 33 FOLLOWER]: Advancing to term 34
I20251028 14:52:14.214510 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:14.214854 3785 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Leader election won for term 34
I20251028 14:52:14.215066 3785 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.215715 3785 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.216723 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.217505 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 34 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.218299 3564 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Rejecting Update request from peer d7cc28c72b6e454f98cc0096e3c57a1d for earlier term 33. Current term is 34. Ops: [1.1-33.12]
W20251028 14:52:14.219805 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.220270 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 34 candidate_status { last_received { term: 32 index: 11 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.222469 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 34 candidate_status { last_received { term: 32 index: 11 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.222744 3427 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 LEADER]: Stepping down as leader of term 33
I20251028 14:52:14.222841 3427 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.223150 3427 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 33.12, Last appended by leader: 12, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.223367 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 33 FOLLOWER]: Advancing to term 34
I20251028 14:52:14.227566 3427 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 34 because replica has last-logged OpId of term: 33 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 11.
I20251028 14:52:14.229753 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 32 to 34, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 34 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.232126 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
W20251028 14:52:14.233268 3842 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.233400 3781 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20251028 14:52:14.234855 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.236032 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.236702 3564 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 32 index: 11. Preceding OpId from leader: term: 34 index: 12. (index mismatch)
I20251028 14:52:14.237720 3774 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:14.237939 3774 consensus_queue.cc:1243] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 32.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251028 14:52:14.239466 3564 pending_rounds.cc:85] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 0
W20251028 14:52:14.239591 3830 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.239681 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20251028 14:52:14.239774 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.239820 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.239936 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20251028 14:52:14.240011 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240077 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20251028 14:52:14.240150 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240211 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 9.4
I20251028 14:52:14.240278 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240339 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 12.5
I20251028 14:52:14.240331 3705 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 34 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 34 index: 12. (index mismatch)
I20251028 14:52:14.240397 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240468 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 17.6
I20251028 14:52:14.240597 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240676 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 20.7
I20251028 14:52:14.240741 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240823 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 23.8
I20251028 14:52:14.240891 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240952 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 26.9
I20251028 14:52:14.241011 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.240929 3774 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:14.241070 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 29.10
I20251028 14:52:14.241133 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.241189 3564 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 32.11
I20251028 14:52:14.241271 3564 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251028 14:52:14.241736 3842 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.249593 3814 mvcc.cc:204] Tried to move back new op lower bound from 7215772197823811584 to 7215772196647514112. Current Snapshot: MvccSnapshot[applied={T|T < 7215772197823811584}]
W20251028 14:52:14.257750 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.262084 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.262274 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Advancing to term 35
I20251028 14:52:14.264736 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.265729 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 35 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.266466 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.267033 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.267256 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 LEADER]: Stepping down as leader of term 34
I20251028 14:52:14.267354 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
W20251028 14:52:14.268149 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.270313 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.270743 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.271867 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 34.16, Last appended by leader: 16, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.272615 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Advancing to term 35
I20251028 14:52:14.275878 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 35 because replica has last-logged OpId of term: 34 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.276512 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.277232 3781 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20251028 14:52:14.281736 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.286397 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.286859 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:14.288254 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.288491 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.288621 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 33 FOLLOWER]: Advancing to term 34
W20251028 14:52:14.291126 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.291436 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.292424 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 34 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
W20251028 14:52:14.292425 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.293365 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 34 candidate_status { last_received { term: 33 index: 12 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.293620 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 34 candidate_status { last_received { term: 33 index: 12 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.293895 3564 raft_consensus.cc:2393] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c in current term 34: Already voted for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in this term.
I20251028 14:52:14.293916 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.294118 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.294219 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 34 FOLLOWER]: Advancing to term 35
I20251028 14:52:14.296339 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.296743 3427 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 34.
I20251028 14:52:14.297221 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 35 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.297610 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:14.297955 3857 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 34 FOLLOWER]: Leader election won for term 34
I20251028 14:52:14.298017 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.298102 3857 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 34 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.298321 3564 raft_consensus.cc:2393] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c in current term 35: Already voted for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in this term.
I20251028 14:52:14.298655 3857 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 33.12, Last appended by leader: 12, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.304301 3564 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 32 index: 11. Preceding OpId from leader: term: 34 index: 14. (index mismatch)
I20251028 14:52:14.304804 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 33 to 34, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 34 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:14.304986 3776 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:14.305173 3776 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 32.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
W20251028 14:52:14.304049 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.301440 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 35 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.304844 3428 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 33 index: 12. Preceding OpId from leader: term: 34 index: 14. (index mismatch)
I20251028 14:52:14.306506 3776 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:14.308673 3427 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 35.
I20251028 14:52:14.309640 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:14.310511 3776 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 35 FOLLOWER]: Leader election won for term 35
I20251028 14:52:14.312331 3564 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 0
I20251028 14:52:14.312433 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20251028 14:52:14.312511 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.312597 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 5.2
I20251028 14:52:14.312685 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.312768 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 8.3
I20251028 14:52:14.312844 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.312912 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 11.4
I20251028 14:52:14.312978 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.313169 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 14.5
I20251028 14:52:14.313313 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.313401 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 17.6
I20251028 14:52:14.313479 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.313570 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 20.7
I20251028 14:52:14.313647 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.313719 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 23.8
I20251028 14:52:14.313796 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.313868 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 26.9
I20251028 14:52:14.313961 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.314030 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 29.10
I20251028 14:52:14.314095 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.314160 3564 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 32.11
I20251028 14:52:14.314224 3564 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.316744 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:14.317725 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.318019 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.318176 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 34 FOLLOWER]: Advancing to term 35
I20251028 14:52:14.321419 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.323400 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.323643 3705 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 34 LEADER]: Stepping down as leader of term 34
I20251028 14:52:14.323745 3705 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 34 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.324080 3705 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 34.15, Last appended by leader: 15, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.324370 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 34 FOLLOWER]: Advancing to term 35
I20251028 14:52:14.324424 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 35 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.326215 3427 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 34. Current term is 35. Ops: []
I20251028 14:52:14.326598 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.326820 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.326927 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Advancing to term 36
I20251028 14:52:14.328224 3776 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 35 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.328609 3776 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 34.16, Last appended by leader: 16, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.329756 3705 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 35 because replica has last-logged OpId of term: 34 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.330718 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.332048 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.332257 3704 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 35 LEADER]: Stepping down as leader of term 35
I20251028 14:52:14.332352 3704 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 35 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.332638 3704 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 35.17, Last appended by leader: 17, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.332927 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 35 FOLLOWER]: Advancing to term 36
I20251028 14:52:14.333648 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 35 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.333861 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 34 FOLLOWER]: Advancing to term 35
I20251028 14:52:14.333493 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 34 to 35, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 35 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:14.336593 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 36 candidate_status { last_received { term: 34 index: 16 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.336733 3704 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 36 because replica has last-logged OpId of term: 35 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 16.
I20251028 14:52:14.336798 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 35 FOLLOWER]: Advancing to term 36
I20251028 14:52:14.337466 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 36 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.339182 3563 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 36.
I20251028 14:52:14.340234 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 35 because replica has last-logged OpId of term: 34 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.340283 3353 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: d400e7c613e146ab91729d201413b76c
I20251028 14:52:14.340811 3353 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c
I20251028 14:52:14.341676 3774 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 FOLLOWER]: Leader election won for term 36
I20251028 14:52:14.341861 3774 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.342119 3774 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 34.16, Last appended by leader: 16, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.343094 3774 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20251028 14:52:14.345487 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.345849 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 35 to 36, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 36 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:14.347361 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.361243 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.361848 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:14.362644 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.362862 3564 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.362979 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 35 FOLLOWER]: Advancing to term 36
W20251028 14:52:14.364531 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.366477 3564 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.367240 3564 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 36 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.368192 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 36 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.368310 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.368528 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.368570 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 35 FOLLOWER]: Advancing to term 36
I20251028 14:52:14.368649 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 FOLLOWER]: Advancing to term 37
I20251028 14:52:14.371852 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 36.
I20251028 14:52:14.371889 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.372714 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:14.372803 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 37 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.373239 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 FOLLOWER]: Leader election won for term 36
I20251028 14:52:14.373435 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.373718 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 34.15, Last appended by leader: 15, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.378687 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
W20251028 14:52:14.379652 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.382177 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.382747 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.382926 3704 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 36 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.383035 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 36 FOLLOWER]: Advancing to term 37
I20251028 14:52:14.387089 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 36 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.387096 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 37 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.387326 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 36 FOLLOWER]: Advancing to term 37
I20251028 14:52:14.387326 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 35 FOLLOWER]: Advancing to term 36
I20251028 14:52:14.388567 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 37 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.388932 3427 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 LEADER]: Stepping down as leader of term 36
I20251028 14:52:14.389091 3427 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.389879 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 34 to 36, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 36 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.390264 3428 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 36.
I20251028 14:52:14.390354 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 37 because replica has last-logged OpId of term: 35 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.390713 3427 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 36.17, Last appended by leader: 17, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.391111 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 FOLLOWER]: Advancing to term 37
W20251028 14:52:14.395233 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.396119 3704 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 37 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.397431 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 37 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.397744 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 36 FOLLOWER]: Advancing to term 37
I20251028 14:52:14.398177 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 37 because replica has last-logged OpId of term: 36 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.400142 3426 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 37.
I20251028 14:52:14.400641 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:14.400981 3776 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 37 FOLLOWER]: Leader election won for term 37
I20251028 14:52:14.401111 3776 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 37 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.401410 3776 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 34.15, Last appended by leader: 15, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.401780 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.402990 3799 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20251028 14:52:14.403239 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.404140 3704 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 37 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.404659 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 37 candidate_status { last_received { term: 34 index: 15 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.404862 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 LEADER]: Stepping down as leader of term 36
I20251028 14:52:14.404151 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 36 to 37, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 37 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:14.404958 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.405230 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 36.16, Last appended by leader: 16, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.405364 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.405543 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 36 FOLLOWER]: Advancing to term 37
I20251028 14:52:14.405578 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 37 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.405682 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 37 FOLLOWER]: Advancing to term 38
I20251028 14:52:14.408016 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 37 because replica has last-logged OpId of term: 36 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 15.
I20251028 14:52:14.408074 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 38 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.408751 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 38 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.409351 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 38 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.409580 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 37 FOLLOWER]: Advancing to term 38
I20251028 14:52:14.409857 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 38 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.410056 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: Advancing to term 38
I20251028 14:52:14.412137 3428 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 38 because replica has last-logged OpId of term: 36 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 35 index: 17.
W20251028 14:52:14.414096 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.414373 3563 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 38.
I20251028 14:52:14.414819 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.415164 3857 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 38 FOLLOWER]: Leader election won for term 38
I20251028 14:52:14.415359 3857 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 38 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.415629 3857 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 35.17, Last appended by leader: 17, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.418044 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 36 to 38, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 38 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:14.419912 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.430735 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:14.431627 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.431880 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 37 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.432030 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 37 FOLLOWER]: Advancing to term 38
I20251028 14:52:14.434809 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 38 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.435334 3426 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 38 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 37. Current term is 38. Ops: []
I20251028 14:52:14.435575 3563 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 36 index: 16. Preceding OpId from leader: term: 37 index: 17. (index mismatch)
I20251028 14:52:14.436062 3857 consensus_queue.cc:1059] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:14.436324 3776 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20251028 14:52:14.436525 3776 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 36.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20251028 14:52:14.436671 3857 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 37 LEADER]: Stepping down as leader of term 37
I20251028 14:52:14.436705 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 38 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.436784 3857 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 37 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.437069 3857 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 37.17, Last appended by leader: 17, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.437417 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 38 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.438011 3857 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 37 FOLLOWER]: Advancing to term 38
I20251028 14:52:14.438180 3562 raft_consensus.cc:1217] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: Deduplicated request from leader. Original: 34.14->[34.15-37.17] Dedup: 34.15->[37.16-37.17]
I20251028 14:52:14.438369 3562 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 15
I20251028 14:52:14.438450 3562 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 36.16
I20251028 14:52:14.438458 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.438565 3562 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.438657 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.438752 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 38 FOLLOWER]: Advancing to term 39
I20251028 14:52:14.441332 3704 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 38 because replica has last-logged OpId of term: 37 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.443039 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 38 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.443109 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.443285 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 37 FOLLOWER]: Advancing to term 38
I20251028 14:52:14.444069 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 39 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.444487 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 39 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.444664 3704 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 38 LEADER]: Stepping down as leader of term 38
I20251028 14:52:14.444761 3704 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 38 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.445046 3704 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 38.18, Last appended by leader: 18, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.445387 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 38 FOLLOWER]: Advancing to term 39
W20251028 14:52:14.445865 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.447881 3704 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 39 because replica has last-logged OpId of term: 38 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 17.
I20251028 14:52:14.449077 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 39 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.449216 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 38 because replica has last-logged OpId of term: 37 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.449294 3562 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 38 FOLLOWER]: Advancing to term 39
I20251028 14:52:14.449867 3353 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c
I20251028 14:52:14.450385 3785 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20251028 14:52:14.451611 3562 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 39.
I20251028 14:52:14.452107 3353 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: d400e7c613e146ab91729d201413b76c
I20251028 14:52:14.452493 3785 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 FOLLOWER]: Leader election won for term 39
I20251028 14:52:14.452667 3785 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.452936 3785 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 36.17, Last appended by leader: 17, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.455463 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 38 to 39, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 39 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:14.466791 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.486392 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.488667 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:14.489537 3562 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.489979 3562 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.490186 3562 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 38 FOLLOWER]: Advancing to term 39
I20251028 14:52:14.492916 3562 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.494100 3562 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 39 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.494866 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 39 candidate_status { last_received { term: 37 index: 17 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.494947 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 39 candidate_status { last_received { term: 37 index: 17 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.495213 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 38 FOLLOWER]: Advancing to term 39
I20251028 14:52:14.495371 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 38 FOLLOWER]: Advancing to term 39
I20251028 14:52:14.495836 3562 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.496086 3562 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.496244 3562 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 FOLLOWER]: Advancing to term 40
I20251028 14:52:14.498792 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 39.
I20251028 14:52:14.499464 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:14.499397 3562 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 40 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.499508 3428 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 39.
I20251028 14:52:14.499809 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 FOLLOWER]: Leader election won for term 39
I20251028 14:52:14.500022 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.500133 3562 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 40 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.500365 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 37.17, Last appended by leader: 17, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.502024 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 40 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.502290 3428 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 LEADER]: Stepping down as leader of term 39
I20251028 14:52:14.502315 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 40 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.502388 3428 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.502542 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 39 FOLLOWER]: Advancing to term 40
I20251028 14:52:14.502712 3428 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 39.18, Last appended by leader: 18, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.503005 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 FOLLOWER]: Advancing to term 40
I20251028 14:52:14.503330 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 37 to 39, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 39 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.505523 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 40 because replica has last-logged OpId of term: 38 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.506892 3428 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 40 because replica has last-logged OpId of term: 39 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.507797 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.508193 3781 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20251028 14:52:14.513487 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:14.513645 3704 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 39 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 37 index: 17. Preceding OpId from leader: term: 39 index: 19. (index mismatch)
I20251028 14:52:14.513823 3428 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 34 index: 14. Preceding OpId from leader: term: 39 index: 19. (index mismatch)
I20251028 14:52:14.514249 3781 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.000s
I20251028 14:52:14.514503 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.514688 3704 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 39 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.514811 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 39 FOLLOWER]: Advancing to term 40
I20251028 14:52:14.514789 3799 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:14.518680 3704 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.519199 3705 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Rejecting Update request from peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 for earlier term 39. Current term is 40. Ops: [39.18-39.19]
W20251028 14:52:14.519903 3824 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.520599 3704 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 40 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.521296 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 40 candidate_status { last_received { term: 37 index: 17 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.521696 3562 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 LEADER]: Stepping down as leader of term 39
I20251028 14:52:14.521795 3562 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.521782 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 40 candidate_status { last_received { term: 37 index: 17 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.521924 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.522131 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.522084 3562 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 39.19, Last appended by leader: 19, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.522233 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Advancing to term 41
I20251028 14:52:14.522459 3562 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 39 FOLLOWER]: Advancing to term 40
I20251028 14:52:14.525012 3562 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 40 because replica has last-logged OpId of term: 39 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 17.
I20251028 14:52:14.525511 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.526052 3857 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20251028 14:52:14.526736 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 41 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.527865 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 41 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.527856 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 41 candidate_status { last_received { term: 38 index: 18 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.528060 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 FOLLOWER]: Advancing to term 41
I20251028 14:52:14.530880 3426 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 41 because replica has last-logged OpId of term: 39 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 18.
I20251028 14:52:14.532208 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 41 candidate_status { last_received { term: 38 index: 18 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.532614 3562 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 40 FOLLOWER]: Advancing to term 41
W20251028 14:52:14.534495 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.535553 3562 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 41.
I20251028 14:52:14.535992 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.536553 3857 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 41 FOLLOWER]: Leader election won for term 41
I20251028 14:52:14.536737 3857 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 41 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.537027 3857 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 38.18, Last appended by leader: 18, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.539819 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 39 to 41, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 41 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:14.543869 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.544436 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:14.545320 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.545508 3426 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.545643 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 39 FOLLOWER]: Advancing to term 40
I20251028 14:52:14.547645 3426 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.548311 3426 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 40 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.548926 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.549297 3704 raft_consensus.cc:2393] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d in current term 40: Already voted for candidate d400e7c613e146ab91729d201413b76c in this term.
I20251028 14:52:14.549304 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.549481 3426 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.549587 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: Advancing to term 42
I20251028 14:52:14.551491 3426 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.552436 3426 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 42 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.549080 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 40 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.552603 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 42 candidate_status { last_received { term: 39 index: 18 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.552812 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 FOLLOWER]: Advancing to term 42
W20251028 14:52:14.554906 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.556051 3562 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 40.
I20251028 14:52:14.556172 3564 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 42.
I20251028 14:52:14.556221 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:14.556674 3353 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: d400e7c613e146ab91729d201413b76c
I20251028 14:52:14.557005 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.557179 3564 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 40 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.557186 3785 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 FOLLOWER]: Leader election won for term 40
I20251028 14:52:14.557281 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 40 FOLLOWER]: Advancing to term 41
I20251028 14:52:14.557348 3785 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.557675 3785 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 39.19, Last appended by leader: 19, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.558017 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 42 candidate_status { last_received { term: 39 index: 18 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.558251 3704 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 41 LEADER]: Stepping down as leader of term 41
I20251028 14:52:14.558382 3704 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 41 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.558694 3704 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 41.19, Last appended by leader: 19, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.558794 3353 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:14.559077 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 41 FOLLOWER]: Advancing to term 42
I20251028 14:52:14.560623 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 39 to 40, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 40 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.562244 3704 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 42 because replica has last-logged OpId of term: 41 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 18.
I20251028 14:52:14.562493 3564 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.563367 3564 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 41 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.563896 3785 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 FOLLOWER]: Leader election won for term 42
I20251028 14:52:14.564075 3785 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.564229 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 41 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.564329 3562 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.564415 3426 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 LEADER]: Stepping down as leader of term 40
I20251028 14:52:14.564365 3785 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 39.18, Last appended by leader: 18, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.564492 3562 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 42 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.564527 3426 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.564620 3562 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 42 FOLLOWER]: Advancing to term 43
I20251028 14:52:14.564779 3426 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 40.20, Last appended by leader: 20, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.565049 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 40 FOLLOWER]: Advancing to term 41
I20251028 14:52:14.567685 3562 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 43 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.567960 3426 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 41 because replica has last-logged OpId of term: 40 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 39 index: 19.
I20251028 14:52:14.568455 3562 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 43 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.568504 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 41 to 42, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 42 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.569329 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 43 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.569518 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 42 FOLLOWER]: Advancing to term 43
I20251028 14:52:14.571605 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 43 because replica has last-logged OpId of term: 41 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.572557 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 43 candidate_status { last_received { term: 34 index: 14 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.572841 3426 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 LEADER]: Stepping down as leader of term 42
I20251028 14:52:14.573001 3426 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.573371 3426 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 42.19, Last appended by leader: 19, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.573626 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 FOLLOWER]: Advancing to term 43
I20251028 14:52:14.575917 3426 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 43 because replica has last-logged OpId of term: 42 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 14.
I20251028 14:52:14.576375 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 41 candidate_status { last_received { term: 39 index: 19 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.576488 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.576573 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 40 FOLLOWER]: Advancing to term 41
W20251028 14:52:14.576916 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.577148 3799 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
I20251028 14:52:14.579033 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 41.
I20251028 14:52:14.579552 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.579892 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 FOLLOWER]: Leader election won for term 41
I20251028 14:52:14.580029 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.580288 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 39.19, Last appended by leader: 19, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.582978 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 40 to 41, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 41 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.590567 3704 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 41 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 37 index: 17. Preceding OpId from leader: term: 41 index: 21. (index mismatch)
I20251028 14:52:14.590706 3426 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 40 index: 20. Preceding OpId from leader: term: 41 index: 21. (index mismatch)
I20251028 14:52:14.591291 3781 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 14, Time since last communication: 0.000s
I20251028 14:52:14.591531 3781 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Peer d7cc28c72b6e454f98cc0096e3c57a1d log is divergent from this leader: its last log entry 40.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20251028 14:52:14.591675 3799 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 14, Time since last communication: 0.000s
I20251028 14:52:14.592737 3426 raft_consensus.cc:1217] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: Deduplicated request from leader. Original: 34.14->[34.15-41.21] Dedup: 39.19->[41.20-41.21]
I20251028 14:52:14.593000 3426 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting all ops after (but not including) 19
I20251028 14:52:14.593129 3426 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting uncommitted NO_OP operation due to leader change: 40.20
I20251028 14:52:14.593225 3426 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.606207 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:14.607206 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.607450 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 41 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.607573 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 41 FOLLOWER]: Advancing to term 42
W20251028 14:52:14.614550 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.610752 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 42 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.619127 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.619211 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 42 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.619379 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 41 FOLLOWER]: Advancing to term 42
I20251028 14:52:14.620317 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 42 candidate_status { last_received { term: 41 index: 21 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.620573 3562 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 LEADER]: Stepping down as leader of term 41
I20251028 14:52:14.620654 3562 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.620963 3562 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.23, Last appended by leader: 23, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.621179 3562 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 41 FOLLOWER]: Advancing to term 42
I20251028 14:52:14.622753 3704 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.623059 3704 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 43 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.623461 3704 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 43 FOLLOWER]: Advancing to term 44
I20251028 14:52:14.624428 3562 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 42 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 21.
W20251028 14:52:14.624557 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.627246 3704 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 44 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.627991 3704 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 44 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.628540 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 44 candidate_status { last_received { term: 41 index: 19 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.628595 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 44 candidate_status { last_received { term: 41 index: 19 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.628800 3562 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 43 FOLLOWER]: Advancing to term 44
I20251028 14:52:14.628922 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 FOLLOWER]: Advancing to term 44
I20251028 14:52:14.630334 3426 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 42 because replica has last-logged OpId of term: 41 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 21.
I20251028 14:52:14.630698 3428 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 FOLLOWER]: Rejecting Update request from peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 for earlier term 41. Current term is 42. Ops: [41.23-41.23]
I20251028 14:52:14.632304 3562 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 44.
I20251028 14:52:14.632401 3427 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 44 because replica has last-logged OpId of term: 42 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 19.
I20251028 14:52:14.633080 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:14.633941 3857 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 44 FOLLOWER]: Leader election won for term 44
I20251028 14:52:14.634177 3857 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 44 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.634466 3857 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 41.19, Last appended by leader: 19, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.637405 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d400e7c613e146ab91729d201413b76c; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.637835 3857 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20251028 14:52:14.639631 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 42 to 44, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 44 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:14.645694 3562 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 34 index: 14. Preceding OpId from leader: term: 44 index: 21. (index mismatch)
I20251028 14:52:14.645856 3427 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 42 index: 19. Preceding OpId from leader: term: 44 index: 21. (index mismatch)
I20251028 14:52:14.646494 3857 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 12, Time since last communication: 0.000s
I20251028 14:52:14.646740 3857 consensus_queue.cc:1243] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer d7cc28c72b6e454f98cc0096e3c57a1d log is divergent from this leader: its last log entry 42.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20251028 14:52:14.648206 3426 raft_consensus.cc:1217] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: Deduplicated request from leader. Original: 34.12->[34.13-44.21] Dedup: 34.16->[35.17-44.21]
I20251028 14:52:14.648357 3426 pending_rounds.cc:85] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting all ops after (but not including) 16
I20251028 14:52:14.648438 3426 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting uncommitted NO_OP operation due to leader change: 36.17
I20251028 14:52:14.648501 3426 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.648612 3426 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting uncommitted NO_OP operation due to leader change: 39.18
I20251028 14:52:14.648666 3426 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.648748 3426 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting uncommitted NO_OP operation due to leader change: 42.19
I20251028 14:52:14.648798 3426 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:14.650321 3857 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 12, Time since last communication: 0.000s
W20251028 14:52:14.655787 3819 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.657939 3819 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.659385 3819 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.662225 3815 mvcc.cc:204] Tried to move back new op lower bound from 7215772199500922880 to 7215772198216663040. Current Snapshot: MvccSnapshot[applied={T|T < 7215772199500922880}]
W20251028 14:52:14.696856 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.702114 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.703653 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:14.705705 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.705974 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.706094 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 42 FOLLOWER]: Advancing to term 43
W20251028 14:52:14.708532 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.711963 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.712469 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.713847 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.713850 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 43 candidate_status { last_received { term: 41 index: 22 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.714078 3564 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 42 FOLLOWER]: Advancing to term 43
I20251028 14:52:14.714214 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 42 FOLLOWER]: Advancing to term 43
I20251028 14:52:14.716979 3704 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 43.
I20251028 14:52:14.717216 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 43 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.717377 3564 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 43 because replica has last-logged OpId of term: 41 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 22.
W20251028 14:52:14.724697 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.735275 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:14.736124 3864 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 FOLLOWER]: Leader election won for term 43
I20251028 14:52:14.736577 3864 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.736936 3864 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 41.22, Last appended by leader: 22, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:14.739938 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.740253 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 41 to 43, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 43 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.741714 3427 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.742033 3427 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.742581 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: Advancing to term 45
I20251028 14:52:14.756611 3427 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.757594 3428 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 44. Current term is 45. Ops: [44.32-44.32]
I20251028 14:52:14.758944 3704 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 43 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 41 index: 21. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20251028 14:52:14.759312 3858 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: INVALID_TERM, Last received: 44.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251028 14:52:14.759641 3858 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 44 LEADER]: Stepping down as leader of term 44
I20251028 14:52:14.759743 3858 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 44 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.759996 3858 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 44.35, Last appended by leader: 35, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.760313 3858 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 44 FOLLOWER]: Advancing to term 45
I20251028 14:52:14.760283 3560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.761091 3774 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 14, Time since last communication: 0.000s
I20251028 14:52:14.761778 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 45 candidate_status { last_received { term: 44 index: 31 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.762099 3427 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 45 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
W20251028 14:52:14.763465 3862 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.763650 3564 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 43 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 41 index: 23. Preceding OpId from leader: term: 43 index: 24. (index mismatch)
I20251028 14:52:14.765166 3774 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20251028 14:52:14.765406 3774 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 41.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20251028 14:52:14.767462 3704 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 45 because replica has last-logged OpId of term: 44 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 31.
I20251028 14:52:14.767602 3562 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 22
I20251028 14:52:14.767700 3562 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted WRITE_OP operation due to leader change: 41.23
W20251028 14:52:14.767880 3562 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Aborted: Op aborted by new leader
I20251028 14:52:14.768340 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c
I20251028 14:52:14.768818 3774 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
W20251028 14:52:14.772296 3656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.786959 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.790405 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.798434 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.805011 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.813301 3659 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.813809 3656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.815781 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.818045 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.822055 3656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.823660 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.823755 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:14.824738 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.824965 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 43 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.825101 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 43 FOLLOWER]: Advancing to term 44
I20251028 14:52:14.828245 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.828928 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 44 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.829447 3564 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Rejecting Update request from peer d7cc28c72b6e454f98cc0096e3c57a1d for earlier term 43. Current term is 44. Ops: []
W20251028 14:52:14.829524 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.830214 3831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.830547 3564 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:14.830739 3564 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Starting forced leader election (received explicit request)
W20251028 14:52:14.830744 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.830858 3564 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Advancing to term 45
I20251028 14:52:14.830984 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 44 candidate_status { last_received { term: 43 index: 32 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.831168 3427 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 LEADER]: Stepping down as leader of term 43
I20251028 14:52:14.831239 3427 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:14.831501 3427 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 43.32, Last appended by leader: 32, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.831786 3427 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 43 FOLLOWER]: Advancing to term 44
W20251028 14:52:14.833218 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.833861 3564 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:14.834206 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.834436 3564 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 45 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.834774 3427 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 44.
I20251028 14:52:14.835319 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:14.835637 3860 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Leader election won for term 44
I20251028 14:52:14.835775 3860 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.836041 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 44 candidate_status { last_received { term: 43 index: 32 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.836249 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 43 FOLLOWER]: Advancing to term 44
I20251028 14:52:14.836087 3860 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 43.32, Last appended by leader: 32, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:14.837430 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.837764 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.839066 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 44.
I20251028 14:52:14.839478 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 43 to 44, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 44 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:14.840013 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 45 candidate_status { last_received { term: 44 index: 35 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.840260 3427 raft_consensus.cc:2393] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in current term 45: Already voted for candidate d7cc28c72b6e454f98cc0096e3c57a1d in this term.
I20251028 14:52:14.840967 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 45 candidate_status { last_received { term: 44 index: 35 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
W20251028 14:52:14.844691 3656 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.845391 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.844691 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.846099 3704 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 45.
I20251028 14:52:14.846521 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:14.846817 3860 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: Leader election won for term 45
I20251028 14:52:14.846980 3860 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.847208 3860 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 44.35, Last appended by leader: 34, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:14.849431 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.849478 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 44 to 45, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 45 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:14.850085 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.853490 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:14.855656 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.863921 3427 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 44 index: 31. Preceding OpId from leader: term: 45 index: 37. (index mismatch)
I20251028 14:52:14.864584 3860 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 31, Time since last communication: 0.000s
I20251028 14:52:14.879055 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:14.879276 3704 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 44 index: 35. Preceding OpId from leader: term: 45 index: 37. (index mismatch)
I20251028 14:52:14.880013 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.880201 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 44 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.880193 3799 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 33, Time since last communication: 0.000s
I20251028 14:52:14.880326 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 44 FOLLOWER]: Advancing to term 45
I20251028 14:52:14.883415 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.884526 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 45 candidate_status { last_received { term: 43 index: 32 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.884752 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 44 FOLLOWER]: Advancing to term 45
I20251028 14:52:14.885726 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 45 candidate_status { last_received { term: 43 index: 32 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.885905 3562 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 LEADER]: Stepping down as leader of term 44
I20251028 14:52:14.885990 3562 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.886232 3562 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 44.33, Last appended by leader: 33, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.886484 3562 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 44 FOLLOWER]: Advancing to term 45
I20251028 14:52:14.887588 3428 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 45.
I20251028 14:52:14.888278 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 45 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:14.889077 3858 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Leader election won for term 45
I20251028 14:52:14.889482 3858 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 45 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.889873 3858 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 43.32, Last appended by leader: 32, Current term: 45, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.894852 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 44 to 45, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 45 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:14.896832 3562 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 45 because replica has last-logged OpId of term: 44 index: 33, which is greater than that of the candidate, which has last-logged OpId of term: 43 index: 32.
I20251028 14:52:14.901371 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 45 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
W20251028 14:52:14.904453 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.910072 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:14.910349 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.910485 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Advancing to term 46
I20251028 14:52:14.913172 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 46 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.913753 3704 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 46 FOLLOWER]: Rejecting Update request from peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 for earlier term 45. Current term is 46. Ops: []
W20251028 14:52:14.917400 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.918071 3860 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: INVALID_TERM, Last received: 45.46, Next index: 47, Last known committed idx: 44, Time since last communication: 0.001s
I20251028 14:52:14.918493 3799 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 LEADER]: Stepping down as leader of term 45
I20251028 14:52:14.918623 3799 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:14.918884 3799 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 45.47, Last appended by leader: 47, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.918974 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 46 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:14.919224 3799 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: Advancing to term 46
I20251028 14:52:14.919306 3427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 46 candidate_status { last_received { term: 45 index: 46 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:14.919505 3427 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Advancing to term 46
I20251028 14:52:14.920298 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 46 candidate_status { last_received { term: 45 index: 46 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.922420 3427 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 46.
I20251028 14:52:14.922930 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:14.923475 3858 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 46 FOLLOWER]: Leader election won for term 46
I20251028 14:52:14.923592 3858 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 46 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.923954 3858 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 45.46, Last appended by leader: 46, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:14.924578 3867 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: Replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 46 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
W20251028 14:52:14.924604 3516 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.925485 3867 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.011s ago.
W20251028 14:52:14.926016 3516 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:14.926461 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 45 to 46, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 46 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:14.926966 3516 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.929221 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.930027 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.930447 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:14.933271 3383 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.933307 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:14.934734 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:14.938257 3562 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 44 index: 33. Preceding OpId from leader: term: 45 index: 34. (index mismatch)
I20251028 14:52:14.938326 3428 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 45 index: 46. Preceding OpId from leader: term: 46 index: 48. (index mismatch)
I20251028 14:52:14.938804 3563 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 46 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 45 index: 47. Preceding OpId from leader: term: 46 index: 48. (index mismatch)
I20251028 14:52:14.938820 3872 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 31, Time since last communication: 0.000s
I20251028 14:52:14.939015 3872 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 44.33 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 31
I20251028 14:52:14.939420 3858 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251028 14:52:14.939581 3858 consensus_queue.cc:1243] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 45.47 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 46
I20251028 14:52:14.939726 3857 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20251028 14:52:14.940733 3562 pending_rounds.cc:85] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 46
I20251028 14:52:14.940763 3563 raft_consensus.cc:1217] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: Deduplicated request from leader. Original: 43.31->[43.32-45.36] Dedup: 43.32->[45.33-45.36]
I20251028 14:52:14.940821 3562 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted WRITE_OP operation due to leader change: 45.47
I20251028 14:52:14.940886 3563 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 32
I20251028 14:52:14.940970 3563 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 44.33
I20251028 14:52:14.941048 3563 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251028 14:52:14.941066 3562 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Aborted: Op aborted by new leader
I20251028 14:52:14.942479 3427 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 43 index: 32. Preceding OpId from leader: term: 45 index: 35. (index mismatch)
I20251028 14:52:14.943081 3857 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251028 14:52:14.949689 3865 mvcc.cc:204] Tried to move back new op lower bound from 7215772200707870720 to 7215772200512094208. Current Snapshot: MvccSnapshot[applied={T|T < 7215772200707870720}]
I20251028 14:52:14.976867 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:14.977823 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:14.978057 3428 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:14.978180 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 45 FOLLOWER]: Advancing to term 46
I20251028 14:52:14.990139 3428 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.990798 3428 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 46 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:14.992410 3427 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 45. Current term is 46. Ops: [45.40-45.41]
I20251028 14:52:14.992378 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 46 candidate_status { last_received { term: 45 index: 39 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:14.992584 3705 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 45 LEADER]: Stepping down as leader of term 45
I20251028 14:52:14.992718 3705 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 45 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:14.993407 3859 consensus_queue.cc:1059] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: INVALID_TERM, Last received: 45.39, Next index: 40, Last known committed idx: 38, Time since last communication: 0.000s
I20251028 14:52:14.993769 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 46 candidate_status { last_received { term: 45 index: 39 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:14.993883 3705 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 45.42, Last appended by leader: 42, Current term: 45, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:14.994112 3562 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 45 FOLLOWER]: Advancing to term 46
I20251028 14:52:14.994208 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 45 FOLLOWER]: Advancing to term 46
I20251028 14:52:14.997064 3562 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 46 because replica has last-logged OpId of term: 45 index: 41, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 39.
W20251028 14:52:14.997432 3657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.998342 3705 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 46 because replica has last-logged OpId of term: 45 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 39.
W20251028 14:52:14.998509 3859 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 46 current term: 46 is higher.
W20251028 14:52:14.998505 3657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:14.997978 3428 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:15.000644 3428 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.000782 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Advancing to term 47
I20251028 14:52:15.001336 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 46 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c
I20251028 14:52:15.001770 3864 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Leader election lost for term 46. Reason: could not achieve majority
I20251028 14:52:15.002310 3562 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 46 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 45. Current term is 46. Ops: [45.42-45.42]
W20251028 14:52:15.004173 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.005885 3428 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.006377 3426 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 46. Current term is 47. Ops: [46.59-46.61]
I20251028 14:52:15.007005 3564 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 47 candidate_status { last_received { term: 46 index: 58 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.007211 3872 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: INVALID_TERM, Last received: 46.58, Next index: 59, Last known committed idx: 57, Time since last communication: 0.000s
I20251028 14:52:15.007612 3859 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 46 LEADER]: Stepping down as leader of term 46
I20251028 14:52:15.007712 3859 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 46 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.008064 3859 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 46.61, Last appended by leader: 61, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.008154 3428 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 47 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.008271 3859 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 46 FOLLOWER]: Advancing to term 47
I20251028 14:52:15.008613 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 47 candidate_status { last_received { term: 46 index: 58 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.010972 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 47 because replica has last-logged OpId of term: 46 index: 61, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 58.
I20251028 14:52:15.011607 3355 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c
I20251028 14:52:15.011955 3864 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
W20251028 14:52:15.012405 3657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.013989 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.018170 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.019490 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.019829 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.019795 3843 tablet_replica.cc:1306] Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 21 similar messages]
W20251028 14:52:15.019971 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.025174 3518 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.025739 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.026165 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.027462 3657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.034926 3657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.035157 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.035730 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.036605 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.042407 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.045469 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.047032 3869 tablet_replica.cc:1306] Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 34 similar messages]
W20251028 14:52:15.047148 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.047195 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.053516 3516 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.057070 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.058583 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.058799 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.064316 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.068526 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.068595 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.071343 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.077554 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.079303 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.080998 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.083917 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.089972 3516 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.093482 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:15.094413 3562 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:15.094708 3562 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 46 FOLLOWER]: Starting forced leader election (received explicit request)
W20251028 14:52:15.094887 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.094913 3562 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 46 FOLLOWER]: Advancing to term 47
I20251028 14:52:15.097903 3562 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.098691 3562 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 47 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
W20251028 14:52:15.099102 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.099129 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.099805 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:15.100001 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 46 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.100139 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 46 FOLLOWER]: Advancing to term 47
I20251028 14:52:15.100317 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 47 candidate_status { last_received { term: 45 index: 41 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.100522 3428 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 46 FOLLOWER]: Advancing to term 47
I20251028 14:52:15.100534 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 47 candidate_status { last_received { term: 45 index: 41 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.100721 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 46 FOLLOWER]: Advancing to term 47
I20251028 14:52:15.103621 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.103936 3705 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 47 because replica has last-logged OpId of term: 45 index: 42, which is greater than that of the candidate, which has last-logged OpId of term: 45 index: 41.
W20251028 14:52:15.104310 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.104444 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 47 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.103477 3428 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 47.
I20251028 14:52:15.104897 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 47 candidate_status { last_received { term: 46 index: 61 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.105130 3426 raft_consensus.cc:2393] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in current term 47: Already voted for candidate d7cc28c72b6e454f98cc0096e3c57a1d in this term.
I20251028 14:52:15.105321 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: d400e7c613e146ab91729d201413b76c
I20251028 14:52:15.105640 3860 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 FOLLOWER]: Leader election won for term 47
I20251028 14:52:15.105791 3860 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:15.105921 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 47 candidate_status { last_received { term: 46 index: 61 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.106055 3860 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 45.41, Last appended by leader: 41, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.107944 3705 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 47.
I20251028 14:52:15.108354 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:15.108359 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 45 to 47, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 47 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:15.108639 3860 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 FOLLOWER]: Leader election won for term 47
I20251028 14:52:15.108795 3860 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:15.109030 3860 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 46.61, Last appended by leader: 61, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:15.112917 3660 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.114073 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.118641 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 46 to 47, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 47 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:15.119534 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.124941 3426 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 45 index: 39. Preceding OpId from leader: term: 47 index: 43. (index mismatch)
I20251028 14:52:15.125032 3705 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Refusing update from remote peer 27f3e1cdaf5f4d6c957b5376bcbde0e5: Log matching property violated. Preceding OpId in replica: term: 45 index: 42. Preceding OpId from leader: term: 47 index: 43. (index mismatch)
I20251028 14:52:15.125762 3799 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 38, Time since last communication: 0.000s
I20251028 14:52:15.126353 3799 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
I20251028 14:52:15.126526 3799 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Peer d400e7c613e146ab91729d201413b76c log is divergent from this leader: its last log entry 45.42 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 41
W20251028 14:52:15.126510 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.127429 3705 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: Aborting all ops after (but not including) 41
I20251028 14:52:15.127616 3705 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c: Aborting uncommitted WRITE_OP operation due to leader change: 45.42
W20251028 14:52:15.127831 3705 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Aborted: Op aborted by new leader
I20251028 14:52:15.195145 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:15.196116 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:15.196365 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.196482 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Advancing to term 48
I20251028 14:52:15.202292 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.203891 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 48 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:15.204576 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 48 candidate_status { last_received { term: 47 index: 50 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.204820 3563 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 LEADER]: Stepping down as leader of term 47
I20251028 14:52:15.204912 3563 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:15.205179 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 48 candidate_status { last_received { term: 47 index: 50 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.205161 3563 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 47.51, Last appended by leader: 51, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.205379 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Advancing to term 48
I20251028 14:52:15.205400 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 FOLLOWER]: Advancing to term 48
I20251028 14:52:15.208516 3426 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 48.
I20251028 14:52:15.208634 3563 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 48 because replica has last-logged OpId of term: 47 index: 51, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 50.
I20251028 14:52:15.209136 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5
I20251028 14:52:15.209582 3872 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 48 FOLLOWER]: Leader election won for term 48
I20251028 14:52:15.209757 3872 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 48 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.209864 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:15.210078 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.210196 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 47 FOLLOWER]: Advancing to term 48
I20251028 14:52:15.210103 3872 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 47.50, Last appended by leader: 50, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.213447 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 47 to 48, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 48 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:15.213737 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.214915 3428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 48 candidate_status { last_received { term: 46 index: 61 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.215090 3427 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Rejecting Update request from peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 for earlier term 47. Current term is 48. Ops: []
I20251028 14:52:15.215133 3428 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 47 FOLLOWER]: Advancing to term 48
I20251028 14:52:15.216054 3704 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 48 LEADER]: Rejecting Update request from peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 for earlier term 47. Current term is 48. Ops: []
I20251028 14:52:15.218081 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 48 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:15.218108 3428 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 48.
I20251028 14:52:15.218933 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 48 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:15.219262 3872 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 48 FOLLOWER]: Leader election won for term 48
I20251028 14:52:15.219429 3872 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 48 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.219724 3872 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 46.61, Last appended by leader: 61, Current term: 48, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:15.219990 3543 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.220856 3543 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.221079 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.221668 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 48 candidate_status { last_received { term: 46 index: 61 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.221889 3563 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 LEADER]: Stepping down as leader of term 47
I20251028 14:52:15.221959 3563 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:15.222225 3563 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 47.62, Last appended by leader: 62, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.222433 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 47 FOLLOWER]: Advancing to term 48
I20251028 14:52:15.222898 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 47 to 48, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 48 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:15.225461 3400 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.226205 3563 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 48 because replica has last-logged OpId of term: 47 index: 62, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 61.
W20251028 14:52:15.228299 3400 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.229010 3404 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.240726 3563 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 47 index: 51. Preceding OpId from leader: term: 48 index: 52. (index mismatch)
I20251028 14:52:15.240726 3428 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 47 index: 50. Preceding OpId from leader: term: 48 index: 52. (index mismatch)
I20251028 14:52:15.241338 3872 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20251028 14:52:15.241595 3859 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20251028 14:52:15.241721 3859 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 47.51 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 50
I20251028 14:52:15.242764 3563 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 50
I20251028 14:52:15.242854 3563 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted WRITE_OP operation due to leader change: 47.51
W20251028 14:52:15.243049 3563 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Aborted: Op aborted by new leader
W20251028 14:52:15.252302 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.277282 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.286088 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.296392 3563 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 47 index: 62. Preceding OpId from leader: term: 48 index: 63. (index mismatch)
I20251028 14:52:15.296448 3428 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 46 index: 58. Preceding OpId from leader: term: 48 index: 63. (index mismatch)
I20251028 14:52:15.297102 3859 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 57, Time since last communication: 0.000s
I20251028 14:52:15.297335 3859 consensus_queue.cc:1243] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 47.62 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 57
I20251028 14:52:15.297518 3872 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 62, Last known committed idx: 57, Time since last communication: 0.000s
I20251028 14:52:15.299103 3563 raft_consensus.cc:1217] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: Deduplicated request from leader. Original: 46.57->[46.58-48.63] Dedup: 46.61->[48.62-48.63]
I20251028 14:52:15.299317 3563 pending_rounds.cc:85] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 61
I20251028 14:52:15.299412 3563 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 47.62
I20251028 14:52:15.299485 3563 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:15.302160 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
W20251028 14:52:15.301306 3817 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.305634 3817 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.004s ago.
W20251028 14:52:15.306600 3817 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.005s ago.
I20251028 14:52:15.306519 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:15.306887 3426 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.307068 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Advancing to term 49
W20251028 14:52:15.307548 3817 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.006s ago.
W20251028 14:52:15.308727 3817 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.000s ago.
W20251028 14:52:15.310577 3817 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.002s ago.
W20251028 14:52:15.310977 3817 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.003s ago.
I20251028 14:52:15.311087 3426 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.312184 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 49 candidate_status { last_received { term: 48 index: 55 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.312376 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: Advancing to term 49
I20251028 14:52:15.315199 3563 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 49.
I20251028 14:52:15.323180 3353 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:15.323566 3785 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 FOLLOWER]: Leader election won for term 49
I20251028 14:52:15.323724 3785 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.324034 3785 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 48.55, Last appended by leader: 55, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.326023 3426 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 49 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.327018 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:15.326915 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 48 to 49, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 49 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:15.327250 3426 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.327380 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 48 FOLLOWER]: Advancing to term 49
I20251028 14:52:15.330235 3426 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.330628 3428 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 48. Current term is 49. Ops: [48.67-48.68]
I20251028 14:52:15.331519 3872 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: INVALID_TERM, Last received: 48.66, Next index: 67, Last known committed idx: 65, Time since last communication: 0.000s
I20251028 14:52:15.331816 3872 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 48 LEADER]: Stepping down as leader of term 48
I20251028 14:52:15.331921 3872 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 48 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.331926 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 49 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.332147 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 48 FOLLOWER]: Advancing to term 49
I20251028 14:52:15.332186 3872 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 48.69, Last appended by leader: 69, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.332487 3872 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 48 FOLLOWER]: Advancing to term 49
I20251028 14:52:15.333555 3426 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 49 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.334636 3563 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 49.
I20251028 14:52:15.335139 3353 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:15.335459 3879 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 FOLLOWER]: Leader election won for term 49
I20251028 14:52:15.335613 3879 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.335935 3879 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 48.66, Last appended by leader: 66, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.336074 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 49 candidate_status { last_received { term: 48 index: 55 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.336239 3704 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 48 LEADER]: Stepping down as leader of term 48
I20251028 14:52:15.336326 3704 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 48 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.336572 3704 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 48.59, Last appended by leader: 59, Current term: 48, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.336792 3704 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 48 FOLLOWER]: Advancing to term 49
I20251028 14:52:15.336869 3426 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 LEADER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 48. Current term is 49. Ops: [48.56-48.58]
I20251028 14:52:15.340313 3704 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 49 because replica has last-logged OpId of term: 48 index: 59, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 55.
W20251028 14:52:15.340788 3657 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.340945 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 49 candidate_status { last_received { term: 48 index: 66 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.341173 3703 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 49 because replica has last-logged OpId of term: 48 index: 69, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 66.
W20251028 14:52:15.342238 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.344264 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.345099 3564 raft_consensus.cc:1240] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 48. Current term is 49. Ops: [48.56-48.57]
I20251028 14:52:15.345144 3562 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 48. Current term is 49. Ops: [48.64-48.64]
I20251028 14:52:15.350792 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 48 to 49, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 49 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:15.352108 3563 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 48 index: 63. Preceding OpId from leader: term: 49 index: 68. (index mismatch)
I20251028 14:52:15.355685 3785 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 63, Time since last communication: 0.000s
I20251028 14:52:15.357093 3705 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 49 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 48 index: 69. Preceding OpId from leader: term: 49 index: 68. (term mismatch)
I20251028 14:52:15.357241 3705 pending_rounds.cc:85] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Aborting all ops after (but not including) 67
I20251028 14:52:15.357331 3705 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 48.68
W20251028 14:52:15.357551 3705 tablet_replica.cc:1306] Aborted: Op aborted by new leader: cancelling pending write operations [suppressed 25 similar messages]
W20251028 14:52:15.357710 3705 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Aborted: Op aborted by new leader
I20251028 14:52:15.358116 3705 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Aborting uncommitted WRITE_OP operation due to leader change: 48.69
W20251028 14:52:15.358283 3705 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Aborted: Op aborted by new leader
I20251028 14:52:15.361192 3809 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20251028 14:52:15.361457 3809 consensus_queue.cc:1243] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Peer d400e7c613e146ab91729d201413b76c log is divergent from this leader: its last log entry 48.67 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 66
I20251028 14:52:15.367993 3703 pending_rounds.cc:85] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Aborting all ops after (but not including) 66
I20251028 14:52:15.368114 3703 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 48.67
W20251028 14:52:15.368328 3703 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Aborted: Op aborted by new leader
I20251028 14:52:15.379482 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:15.393963 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:15.394296 3563 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.394438 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Advancing to term 50
W20251028 14:52:15.397365 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.398205 3563 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 50 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.400080 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 50 candidate_status { last_received { term: 48 index: 55 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.400349 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 49 FOLLOWER]: Advancing to term 50
I20251028 14:52:15.404121 3705 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 50 because replica has last-logged OpId of term: 48 index: 59, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 55.
I20251028 14:52:15.407495 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 50 candidate_status { last_received { term: 48 index: 55 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.407931 3426 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 LEADER]: Stepping down as leader of term 49
I20251028 14:52:15.408282 3426 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.408618 3426 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 49.56, Last appended by leader: 56, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.409439 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 FOLLOWER]: Advancing to term 50
I20251028 14:52:15.411015 3563 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 50 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.412158 3563 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:15.412386 3563 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.412511 3563 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 49 FOLLOWER]: Advancing to term 50
I20251028 14:52:15.416414 3426 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 50 because replica has last-logged OpId of term: 49 index: 56, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 55.
I20251028 14:52:15.421380 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:15.421792 3563 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 50 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.421947 3781 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20251028 14:52:15.422184 3564 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 50 FOLLOWER]: Rejecting Update request from peer d7cc28c72b6e454f98cc0096e3c57a1d for earlier term 49. Current term is 50. Ops: [49.73-49.73]
I20251028 14:52:15.423358 3563 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 50 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.426112 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 50 candidate_status { last_received { term: 49 index: 72 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.426590 3809 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: INVALID_TERM, Last received: 49.72, Next index: 73, Last known committed idx: 72, Time since last communication: 0.000s
I20251028 14:52:15.427671 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 50 candidate_status { last_received { term: 49 index: 72 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.427903 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 49 FOLLOWER]: Advancing to term 50
I20251028 14:52:15.429287 3426 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 LEADER]: Stepping down as leader of term 49
I20251028 14:52:15.429442 3426 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.429771 3426 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 49.74, Last appended by leader: 74, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.431047 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 49 FOLLOWER]: Advancing to term 50
I20251028 14:52:15.431717 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 50 because replica has last-logged OpId of term: 49 index: 74, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 72.
W20251028 14:52:15.433269 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.433220 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.434227 3426 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 50 because replica has last-logged OpId of term: 49 index: 74, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 72.
W20251028 14:52:15.434391 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.434736 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
W20251028 14:52:15.434900 3809 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 50 current term: 50 is higher.
I20251028 14:52:15.435263 3781 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
W20251028 14:52:15.435496 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.441469 3404 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.442813 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.443969 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.445290 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.446012 3884 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.446177 3881 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.447360 3882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.449388 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.453506 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.454588 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.455686 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.456943 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.459141 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.459790 3406 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.460320 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.461139 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.461357 3680 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.461416 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.462450 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.461632 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.464957 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.465088 3882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.465194 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.465843 3881 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.468010 3881 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.468538 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.469887 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.469906 3543 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.471128 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.471944 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.472704 3543 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.473214 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.475296 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.476840 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.477715 3850 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.478839 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.479290 3854 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.479552 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.480392 3404 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.481235 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.482734 3406 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.483253 3886 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.483558 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.483695 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.484505 3881 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.484642 3406 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.485664 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:15.486487 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:15.486704 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 50 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.486846 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 50 FOLLOWER]: Advancing to term 51
W20251028 14:52:15.488075 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.488155 3543 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.489007 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.490305 3543 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.490712 3542 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.491032 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.491196 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 51 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:15.491336 3541 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.491961 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 51 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
W20251028 14:52:15.491969 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.492388 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 51 candidate_status { last_received { term: 48 index: 59 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.492637 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 50 FOLLOWER]: Advancing to term 51
W20251028 14:52:15.493175 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.493273 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 51 candidate_status { last_received { term: 48 index: 59 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
W20251028 14:52:15.493371 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.493472 3562 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 50 FOLLOWER]: Advancing to term 51
I20251028 14:52:15.495749 3426 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 51 because replica has last-logged OpId of term: 49 index: 56, which is greater than that of the candidate, which has last-logged OpId of term: 48 index: 59.
W20251028 14:52:15.496243 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.496297 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:15.496443 3562 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 51.
I20251028 14:52:15.496507 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 50 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.496615 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 50 FOLLOWER]: Advancing to term 51
I20251028 14:52:15.497975 3631 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:15.498323 3859 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 51 FOLLOWER]: Leader election won for term 51
I20251028 14:52:15.498463 3859 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 51 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.498742 3859 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 48.59, Last appended by leader: 59, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
W20251028 14:52:15.499464 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.499521 3680 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.500275 3678 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.501375 3261 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 49 to 51, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 51 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:15.502370 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 51 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.503373 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 51 candidate_status { last_received { term: 49 index: 74 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.503633 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 50 FOLLOWER]: Advancing to term 51
W20251028 14:52:15.506484 3680 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.507480 3881 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.508237 3428 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 49 index: 56. Preceding OpId from leader: term: 51 index: 61. (index mismatch)
I20251028 14:52:15.508273 3562 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 51 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 48 index: 55. Preceding OpId from leader: term: 51 index: 61. (index mismatch)
I20251028 14:52:15.508656 3426 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 51.
I20251028 14:52:15.508884 3872 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 55, Time since last communication: 0.000s
W20251028 14:52:15.510151 3886 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.510895 3882 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.511519 3406 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.512203 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.512569 3872 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 60, Last known committed idx: 55, Time since last communication: 0.000s
I20251028 14:52:15.512797 3872 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer d7cc28c72b6e454f98cc0096e3c57a1d log is divergent from this leader: its last log entry 49.56 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 55
W20251028 14:52:15.513890 3406 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.514158 3633 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:15.514464 3872 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 51 FOLLOWER]: Leader election won for term 51
I20251028 14:52:15.514635 3872 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 51 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.515331 3872 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 49.74, Last appended by leader: 74, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.514137 3428 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting all ops after (but not including) 55
I20251028 14:52:15.515671 3428 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting uncommitted NO_OP operation due to leader change: 49.56
I20251028 14:52:15.515755 3428 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:15.519215 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 51 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
W20251028 14:52:15.519815 3404 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.521494 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.522974 3841 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.523126 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.528594 3869 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.530637 3562 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 51 candidate_status { last_received { term: 49 index: 74 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.531073 3562 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 50 FOLLOWER]: Advancing to term 51
W20251028 14:52:15.531390 3541 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.531769 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.532832 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.533030 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.534232 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.534446 3562 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 51.
W20251028 14:52:15.535430 3843 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:15.538231 3870 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:15.538460 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:15.538990 3261 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 49 to 51, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 51 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:15.543352 3404 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:15.553033 3426 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 49 index: 74. Preceding OpId from leader: term: 51 index: 77. (index mismatch)
I20251028 14:52:15.555701 3776 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
I20251028 14:52:15.561285 3562 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 51 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 49 index: 72. Preceding OpId from leader: term: 51 index: 77. (index mismatch)
I20251028 14:52:15.562017 3872 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 72, Time since last communication: 0.000s
I20251028 14:52:15.611232 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:15.612074 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:15.612367 3426 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.612542 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Advancing to term 52
I20251028 14:52:15.615624 3426 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.621716 3426 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 52 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.622519 3563 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 52 candidate_status { last_received { term: 51 index: 69 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.622870 3563 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 51 FOLLOWER]: Advancing to term 52
I20251028 14:52:15.623191 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:15.623448 3426 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.623594 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 51 FOLLOWER]: Advancing to term 52
I20251028 14:52:15.626981 3426 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.627903 3426 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 52 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.629900 3561 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 52 candidate_status { last_received { term: 51 index: 92 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.631347 3561 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 51 FOLLOWER]: Advancing to term 52
I20251028 14:52:15.639855 3561 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 52.
I20251028 14:52:15.640232 3560 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 51. Current term is 52. Ops: [51.86-51.95]
I20251028 14:52:15.628844 3563 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 52.
I20251028 14:52:15.642243 3353 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:15.642527 3426 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 51. Current term is 52. Ops: [51.93-51.93]
I20251028 14:52:15.642592 3353 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:15.642746 3879 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 FOLLOWER]: Leader election won for term 52
I20251028 14:52:15.642897 3879 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.643153 3879 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 51.92, Last appended by leader: 92, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.643255 3864 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 FOLLOWER]: Leader election won for term 52
I20251028 14:52:15.643411 3864 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.643661 3864 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 51.69, Last appended by leader: 69, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.645424 3703 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 52 candidate_status { last_received { term: 51 index: 69 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.645879 3703 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 51 LEADER]: Stepping down as leader of term 51
I20251028 14:52:15.645978 3703 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 51 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.646565 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 51 to 52, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 52 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:15.646808 3703 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 51.69, Last appended by leader: 69, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.647135 3703 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 51 FOLLOWER]: Advancing to term 52
I20251028 14:52:15.647148 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 51 to 52, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 52 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:15.647925 3872 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: INVALID_TERM, Last received: 51.85, Next index: 86, Last known committed idx: 84, Time since last communication: 0.007s
I20251028 14:52:15.651438 3563 raft_consensus.cc:1240] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Rejecting Update request from peer d400e7c613e146ab91729d201413b76c for earlier term 51. Current term is 52. Ops: [51.86-51.96]
W20251028 14:52:15.651700 3659 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.652360 3859 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: INVALID_TERM, Last received: 51.92, Next index: 93, Last known committed idx: 89, Time since last communication: 0.000s
I20251028 14:52:15.654105 3776 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 51 LEADER]: Stepping down as leader of term 51
I20251028 14:52:15.654054 3704 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 52 candidate_status { last_received { term: 51 index: 92 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.654201 3776 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 51 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
W20251028 14:52:15.654743 3675 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.655310 3679 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.655879 3859 consensus_queue.cc:1059] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: INVALID_TERM, Last received: 51.85, Next index: 86, Last known committed idx: 84, Time since last communication: 0.003s
W20251028 14:52:15.657790 3821 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
W20251028 14:52:15.658110 3868 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.660691 3776 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 51.96, Last appended by leader: 96, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.661154 3776 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 51 FOLLOWER]: Advancing to term 52
W20251028 14:52:15.668308 3815 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: Replica d400e7c613e146ab91729d201413b76c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 52 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } } }
W20251028 14:52:15.669024 3815 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:50293: Illegal state: Cannot bump the last assigned timestamp. Tablet is not in leader mode. Last heard from a leader: 0.020s ago.
I20251028 14:52:15.670111 3703 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 52.
I20251028 14:52:15.670387 3704 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 52 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 51 index: 69. Preceding OpId from leader: term: 52 index: 72. (index mismatch)
I20251028 14:52:15.670650 3563 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Refusing update from remote peer d7cc28c72b6e454f98cc0096e3c57a1d: Log matching property violated. Preceding OpId in replica: term: 51 index: 69. Preceding OpId from leader: term: 52 index: 72. (index mismatch)
I20251028 14:52:15.671543 3785 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
W20251028 14:52:15.672022 3889 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:54246: Illegal state: replica d400e7c613e146ab91729d201413b76c is not leader of this config: current role FOLLOWER
I20251028 14:52:15.672035 3809 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20251028 14:52:15.678843 3861 mvcc.cc:204] Tried to move back new op lower bound from 7215772203707457536 to 7215772203598458880. Current Snapshot: MvccSnapshot[applied={T|T < 7215772203235749888 or (T in {7215772203675627520})}]
I20251028 14:52:15.794580 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:15.795792 3560 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:15.796041 3560 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.796176 3560 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Advancing to term 53
I20251028 14:52:15.798792 3560 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.799687 3560 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 53 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.800383 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 53 candidate_status { last_received { term: 52 index: 75 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.800500 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 53 candidate_status { last_received { term: 52 index: 75 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.800613 3426 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 LEADER]: Stepping down as leader of term 52
I20251028 14:52:15.800685 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 52 FOLLOWER]: Advancing to term 53
I20251028 14:52:15.800716 3426 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.801002 3426 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 52.75, Last appended by leader: 75, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.801404 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 FOLLOWER]: Advancing to term 53
I20251028 14:52:15.801381 3560 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:15.801605 3560 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.801718 3560 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 52 FOLLOWER]: Advancing to term 53
I20251028 14:52:15.804136 3705 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 53.
I20251028 14:52:15.804325 3426 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 53.
I20251028 14:52:15.804670 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:15.805037 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 FOLLOWER]: Leader election won for term 53
I20251028 14:52:15.805205 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:15.805291 3560 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.805541 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 52.75, Last appended by leader: 75, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.806022 3560 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 53 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:15.806481 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 53 candidate_status { last_received { term: 51 index: 85 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.806556 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 53 candidate_status { last_received { term: 51 index: 85 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:15.806685 3426 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 LEADER]: Stepping down as leader of term 52
I20251028 14:52:15.806770 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 52 FOLLOWER]: Advancing to term 53
I20251028 14:52:15.806789 3426 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:15.807075 3426 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 52.93, Last appended by leader: 93, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.807344 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 52 FOLLOWER]: Advancing to term 53
I20251028 14:52:15.808557 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 52 to 53, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 53 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:15.809544 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 53 because replica has last-logged OpId of term: 51 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 85.
I20251028 14:52:15.810432 3426 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 53 because replica has last-logged OpId of term: 52 index: 93, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 85.
I20251028 14:52:15.811010 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 53 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:15.811868 3799 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 FOLLOWER]: Leader election lost for term 53. Reason: could not achieve majority
I20251028 14:52:15.939747 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:15.940984 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:15.941299 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 53 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.941442 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 53 FOLLOWER]: Advancing to term 54
I20251028 14:52:15.943527 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 54 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.944262 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 54 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:15.944888 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 54 candidate_status { last_received { term: 52 index: 75 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.945097 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 53 FOLLOWER]: Advancing to term 54
I20251028 14:52:15.945487 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:15.945675 3560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 54 candidate_status { last_received { term: 52 index: 75 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.945793 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 53 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:15.945879 3560 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 LEADER]: Stepping down as leader of term 53
I20251028 14:52:15.945919 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 53 FOLLOWER]: Advancing to term 54
I20251028 14:52:15.945991 3560 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:15.946275 3560 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 53.76, Last appended by leader: 76, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.946573 3560 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 FOLLOWER]: Advancing to term 54
I20251028 14:52:15.947691 3426 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 54.
I20251028 14:52:15.948271 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:15.948616 3776 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 54 FOLLOWER]: Leader election won for term 54
I20251028 14:52:15.948771 3776 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 54 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.949085 3776 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 52.75, Last appended by leader: 75, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.949908 3560 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 54 because replica has last-logged OpId of term: 53 index: 76, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 75.
I20251028 14:52:15.949064 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 54 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.951637 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 54 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:15.952361 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 54 candidate_status { last_received { term: 51 index: 96 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:15.952409 3560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 54 candidate_status { last_received { term: 51 index: 96 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:15.952600 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 53 FOLLOWER]: Advancing to term 54
I20251028 14:52:15.952672 3560 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 53 FOLLOWER]: Advancing to term 54
I20251028 14:52:15.952699 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 53 to 54, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 54 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:15.955588 3560 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 54.
I20251028 14:52:15.955737 3426 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 54 because replica has last-logged OpId of term: 52 index: 93, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 96.
I20251028 14:52:15.956471 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters: d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:15.956838 3776 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 54 FOLLOWER]: Leader election won for term 54
I20251028 14:52:15.957001 3776 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 54 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:15.957295 3776 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 51.96, Last appended by leader: 96, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:15.959645 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 52 to 54, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 54 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:16.058291 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 0
I20251028 14:52:16.059507 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:16.059769 3426 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 54 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:16.059919 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 54 FOLLOWER]: Advancing to term 55
I20251028 14:52:16.061923 3426 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.062685 3426 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 55 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:16.063223 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 55 candidate_status { last_received { term: 52 index: 75 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:16.063203 3560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 55 candidate_status { last_received { term: 52 index: 75 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:16.063424 3560 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 54 FOLLOWER]: Advancing to term 55
I20251028 14:52:16.063424 3705 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 54 LEADER]: Stepping down as leader of term 54
I20251028 14:52:16.063526 3705 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 54 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:16.063638 3426 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
from {username='slave'} at 127.0.0.1:47500
I20251028 14:52:16.063848 3426 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 54 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:16.063850 3705 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 54.76, Last appended by leader: 76, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.063956 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 54 FOLLOWER]: Advancing to term 55
I20251028 14:52:16.064126 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 54 FOLLOWER]: Advancing to term 55
I20251028 14:52:16.066529 3560 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 55 because replica has last-logged OpId of term: 53 index: 76, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 75.
I20251028 14:52:16.067045 3705 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 55 because replica has last-logged OpId of term: 54 index: 76, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 75.
I20251028 14:52:16.067153 3426 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.067541 3355 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7cc28c72b6e454f98cc0096e3c57a1d; no voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c
I20251028 14:52:16.067883 3426 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 55 election: Requested vote from peers 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:16.068182 3892 raft_consensus.cc:2749] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20251028 14:52:16.068397 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 55 candidate_status { last_received { term: 52 index: 93 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:16.068315 3560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" candidate_term: 55 candidate_status { last_received { term: 52 index: 93 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:16.068580 3705 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 54 LEADER]: Stepping down as leader of term 54
I20251028 14:52:16.068681 3705 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 54 LEADER]: Becoming Follower/Learner. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:16.068670 3560 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 54 FOLLOWER]: Advancing to term 55
I20251028 14:52:16.068938 3705 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 54.97, Last appended by leader: 97, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.069176 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 54 FOLLOWER]: Advancing to term 55
I20251028 14:52:16.070827 3560 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7cc28c72b6e454f98cc0096e3c57a1d in term 55.
I20251028 14:52:16.071159 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate d7cc28c72b6e454f98cc0096e3c57a1d for term 55 because replica has last-logged OpId of term: 54 index: 97, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 93.
I20251028 14:52:16.071370 3353 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:16.071723 3892 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 FOLLOWER]: Leader election won for term 55
I20251028 14:52:16.071918 3892 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 LEADER]: Becoming Leader. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:16.072211 3892 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 52.93, Last appended by leader: 93, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.074652 3263 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d reported cstate change: term changed from 54 to 55, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1). New cstate: current_term: 55 leader_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:16.253612 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 1
I20251028 14:52:16.254787 3560 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:16.255077 3560 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 55 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:16.255189 3560 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 55 FOLLOWER]: Advancing to term 56
I20251028 14:52:16.257277 3560 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.257937 3560 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 56 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:16.258519 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 56 candidate_status { last_received { term: 53 index: 76 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:16.258615 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 56 candidate_status { last_received { term: 53 index: 76 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:16.258751 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 FOLLOWER]: Advancing to term 56
I20251028 14:52:16.258792 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 55 FOLLOWER]: Advancing to term 56
I20251028 14:52:16.258978 3560 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
from {username='slave'} at 127.0.0.1:49816
I20251028 14:52:16.259135 3560 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 55 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:16.259254 3560 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 55 FOLLOWER]: Advancing to term 56
I20251028 14:52:16.261094 3560 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.261634 3426 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 in term 56.
I20251028 14:52:16.261732 3560 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 56 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), d400e7c613e146ab91729d201413b76c (127.0.104.3:37725)
I20251028 14:52:16.261773 3705 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 56 because replica has last-logged OpId of term: 54 index: 76, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 76.
I20251028 14:52:16.262142 3492 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:16.262336 3705 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 56 candidate_status { last_received { term: 51 index: 85 } } ignore_live_leader: true dest_uuid: "d400e7c613e146ab91729d201413b76c"
I20251028 14:52:16.262471 3799 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 FOLLOWER]: Leader election won for term 56
I20251028 14:52:16.262521 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 55 FOLLOWER]: Advancing to term 56
I20251028 14:52:16.262624 3799 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 LEADER]: Becoming Leader. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:16.262777 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" candidate_term: 56 candidate_status { last_received { term: 51 index: 85 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:16.262969 3426 raft_consensus.cc:3055] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 LEADER]: Stepping down as leader of term 55
I20251028 14:52:16.263051 3426 raft_consensus.cc:740] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 LEADER]: Becoming Follower/Learner. State: Replica: d7cc28c72b6e454f98cc0096e3c57a1d, State: Running, Role: LEADER
I20251028 14:52:16.263016 3799 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 53.76, Last appended by leader: 76, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.263402 3426 consensus_queue.cc:260] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 55.94, Last appended by leader: 94, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.263680 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 55 FOLLOWER]: Advancing to term 56
I20251028 14:52:16.265512 3705 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 56 because replica has last-logged OpId of term: 54 index: 97, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 85.
I20251028 14:52:16.265658 3263 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 reported cstate change: term changed from 54 to 56, leader changed from d400e7c613e146ab91729d201413b76c (127.0.104.3) to 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2). New cstate: current_term: 56 leader_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:16.266674 3426 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate 27f3e1cdaf5f4d6c957b5376bcbde0e5 for term 56 because replica has last-logged OpId of term: 55 index: 94, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 85.
I20251028 14:52:16.267181 3492 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [CANDIDATE]: Term 56 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5; no voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d
I20251028 14:52:16.267526 3799 raft_consensus.cc:2749] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 FOLLOWER]: Leader election lost for term 56. Reason: could not achieve majority
I20251028 14:52:16.333725 416 txn_write_ops-itest.cc:639] attempting to promote replicas at tserver 2
I20251028 14:52:16.334772 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:16.335052 3705 raft_consensus.cc:493] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 56 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:16.335176 3705 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 56 FOLLOWER]: Advancing to term 57
I20251028 14:52:16.338037 3705 raft_consensus.cc:515] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 57 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.338800 3705 leader_election.cc:290] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 57 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:16.339360 3560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 57 candidate_status { last_received { term: 54 index: 76 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:16.339387 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "253448b9924d42f48cd2cf3e477284c3" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 57 candidate_status { last_received { term: 54 index: 76 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:16.339593 3560 raft_consensus.cc:3055] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 LEADER]: Stepping down as leader of term 56
I20251028 14:52:16.339694 3560 raft_consensus.cc:740] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: 27f3e1cdaf5f4d6c957b5376bcbde0e5, State: Running, Role: LEADER
I20251028 14:52:16.339691 3426 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 56 FOLLOWER]: Advancing to term 57
I20251028 14:52:16.339926 3705 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707"
dest_uuid: "d400e7c613e146ab91729d201413b76c"
from {username='slave'} at 127.0.0.1:54240
I20251028 14:52:16.339982 3560 consensus_queue.cc:260] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 56.77, Last appended by leader: 77, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.340185 3705 raft_consensus.cc:493] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 56 FOLLOWER]: Starting forced leader election (received explicit request)
I20251028 14:52:16.340229 3560 raft_consensus.cc:3060] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 FOLLOWER]: Advancing to term 57
I20251028 14:52:16.340293 3705 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 56 FOLLOWER]: Advancing to term 57
I20251028 14:52:16.341928 3426 raft_consensus.cc:2468] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 57.
I20251028 14:52:16.342615 3633 leader_election.cc:304] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d400e7c613e146ab91729d201413b76c, d7cc28c72b6e454f98cc0096e3c57a1d; no voters:
I20251028 14:52:16.342949 3776 raft_consensus.cc:2804] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 57 FOLLOWER]: Leader election won for term 57
I20251028 14:52:16.343104 3776 raft_consensus.cc:697] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [term 57 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:16.343200 3560 raft_consensus.cc:2410] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 57 because replica has last-logged OpId of term: 56 index: 77, which is greater than that of the candidate, which has last-logged OpId of term: 54 index: 76.
I20251028 14:52:16.343125 3705 raft_consensus.cc:515] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 57 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.343398 3776 consensus_queue.cc:237] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 75, Committed index: 75, Last appended: 54.76, Last appended by leader: 76, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.343979 3705 leader_election.cc:290] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 57 election: Requested vote from peers d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1:39177), 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2:41917)
I20251028 14:52:16.344982 3560 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 57 candidate_status { last_received { term: 54 index: 97 } } ignore_live_leader: true dest_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5"
I20251028 14:52:16.345304 3560 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 56 FOLLOWER]: Advancing to term 57
I20251028 14:52:16.348626 3560 raft_consensus.cc:2468] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d400e7c613e146ab91729d201413b76c in term 57.
I20251028 14:52:16.349164 3631 leader_election.cc:304] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27f3e1cdaf5f4d6c957b5376bcbde0e5, d400e7c613e146ab91729d201413b76c; no voters:
I20251028 14:52:16.349570 3872 raft_consensus.cc:2804] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 57 FOLLOWER]: Leader election won for term 57
I20251028 14:52:16.349742 3872 raft_consensus.cc:697] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [term 57 LEADER]: Becoming Leader. State: Replica: d400e7c613e146ab91729d201413b76c, State: Running, Role: LEADER
I20251028 14:52:16.350044 3872 consensus_queue.cc:237] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 54.97, Last appended by leader: 97, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } }
I20251028 14:52:16.350103 3262 catalog_manager.cc:5649] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 56 to 57, leader changed from 27f3e1cdaf5f4d6c957b5376bcbde0e5 (127.0.104.2) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 57 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:16.353457 3426 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c69e5c1a69cf438180b9f398eeaba707" candidate_uuid: "d400e7c613e146ab91729d201413b76c" candidate_term: 57 candidate_status { last_received { term: 54 index: 97 } } ignore_live_leader: true dest_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d"
I20251028 14:52:16.353691 3426 raft_consensus.cc:3060] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 56 FOLLOWER]: Advancing to term 57
I20251028 14:52:16.355242 3262 catalog_manager.cc:5649] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c reported cstate change: term changed from 55 to 57, leader changed from d7cc28c72b6e454f98cc0096e3c57a1d (127.0.104.1) to d400e7c613e146ab91729d201413b76c (127.0.104.3). New cstate: current_term: 57 leader_uuid: "d400e7c613e146ab91729d201413b76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d400e7c613e146ab91729d201413b76c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37725 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:16.356541 3426 raft_consensus.cc:2410] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate d400e7c613e146ab91729d201413b76c for term 57 because replica has last-logged OpId of term: 55 index: 94, which is greater than that of the candidate, which has last-logged OpId of term: 54 index: 97.
I20251028 14:52:16.806824 3426 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P d7cc28c72b6e454f98cc0096e3c57a1d [term 57 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 52 index: 75. Preceding OpId from leader: term: 57 index: 77. (index mismatch)
I20251028 14:52:16.807684 3872 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
I20251028 14:52:16.820363 3560 raft_consensus.cc:1275] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 57 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 56 index: 77. Preceding OpId from leader: term: 57 index: 77. (term mismatch)
I20251028 14:52:16.820524 3560 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 76
I20251028 14:52:16.820614 3560 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 56.77
I20251028 14:52:16.820690 3560 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 57 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:16.821750 3776 consensus_queue.cc:1048] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 75, Time since last communication: 0.000s
I20251028 14:52:16.822005 3776 consensus_queue.cc:1243] T 253448b9924d42f48cd2cf3e477284c3 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer 27f3e1cdaf5f4d6c957b5376bcbde0e5 log is divergent from this leader: its last log entry 53.76 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 75
I20251028 14:52:16.823047 3560 pending_rounds.cc:85] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting all ops after (but not including) 75
I20251028 14:52:16.823158 3560 pending_rounds.cc:107] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5: Aborting uncommitted NO_OP operation due to leader change: 53.76
I20251028 14:52:16.823248 3560 raft_consensus.cc:2889] T 253448b9924d42f48cd2cf3e477284c3 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 57 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:16.864152 3426 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 57 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 55 index: 94. Preceding OpId from leader: term: 57 index: 98. (index mismatch)
I20251028 14:52:16.864940 3776 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cc28c72b6e454f98cc0096e3c57a1d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 89, Time since last communication: 0.000s
I20251028 14:52:16.865163 3776 consensus_queue.cc:1243] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Peer d7cc28c72b6e454f98cc0096e3c57a1d log is divergent from this leader: its last log entry 55.94 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 89
I20251028 14:52:16.866770 3426 raft_consensus.cc:1217] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 57 FOLLOWER]: Deduplicated request from leader. Original: 51.89->[51.90-57.98] Dedup: 51.92->[51.93-57.98]
I20251028 14:52:16.867043 3426 pending_rounds.cc:85] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting all ops after (but not including) 92
I20251028 14:52:16.867118 3426 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting uncommitted NO_OP operation due to leader change: 52.93
I20251028 14:52:16.867206 3426 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 57 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251028 14:52:16.867295 3426 pending_rounds.cc:107] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d: Aborting uncommitted NO_OP operation due to leader change: 55.94
I20251028 14:52:16.867363 3426 raft_consensus.cc:2889] T c69e5c1a69cf438180b9f398eeaba707 P d7cc28c72b6e454f98cc0096e3c57a1d [term 57 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20251028 14:52:16.870783 3814 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:16.871109 3814 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47518 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=72, attempt_no=1}) took 1206 ms. Trace:
I20251028 14:52:16.871238 3814 rpcz_store.cc:276] 1028 14:52:15.664208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.664259 (+ 51us) service_pool.cc:225] Handling call
1028 14:52:16.871075 (+1206816us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:15.666909 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:16.870413 (+1203504us) op_driver.cc:389] HandleFailure(Illegal state: Transaction 15 already open)
1028 14:52:16.870423 (+ 10us) participant_op.cc:94] Released txn lock
1028 14:52:16.870430 (+ 7us) participant_op.cc:287] FINISH: Op aborted
Related trace 'op':
1028 14:52:16.870709 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER)
1028 14:52:16.870719 (+ 10us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.870725 (+ 6us) write_op.cc:454] Released schema lock
1028 14:52:16.870730 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":126,"txn-participant-registration.run_cpu_time_us":734,"txn-participant-registration.run_wall_time_us":2109,"child_traces":[["op",{"prepare.queue_time_us":70,"child_traces":[["op",{}]]}]]}
W20251028 14:52:16.873406 3814 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: Cannot bump the last assigned timestamp. Tablet is not in leader mode. Last heard from a leader: 0.800s ago.
I20251028 14:52:16.873632 3814 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.1:45449 (ReqId={client: fb71aec3fc3c438fbf5a1ebc1bf1c937, seq_no=80, attempt_no=1}) took 1200 ms. Trace:
I20251028 14:52:16.873735 3814 rpcz_store.cc:276] 1028 14:52:15.672937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.672987 (+ 50us) service_pool.cc:225] Handling call
1028 14:52:16.873578 (+1200591us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:16.873238 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:16.873353 (+ 115us) op_driver.cc:389] HandleFailure(Illegal state: Cannot bump the last assigned timestamp. Tablet is not in leader mode. Last heard from a leader: 0.800s ago.)
1028 14:52:16.873362 (+ 9us) participant_op.cc:94] Released txn lock
1028 14:52:16.873368 (+ 6us) participant_op.cc:287] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":1199946}]]}
W20251028 14:52:16.874722 3814 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.001s ago.
I20251028 14:52:16.874997 3814 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47518 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=77, attempt_no=0}) took 1193 ms. Trace:
I20251028 14:52:16.875157 3814 rpcz_store.cc:276] 1028 14:52:15.681276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.681332 (+ 56us) service_pool.cc:225] Handling call
1028 14:52:16.874958 (+1193626us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:16.874300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c69e5c1a69cf438180b9f398eeaba707
1028 14:52:16.874431 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:16.874435 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:52:16.874438 (+ 3us) tablet.cc:660] Decoding operations
1028 14:52:16.874522 (+ 84us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:52:16.874583 (+ 61us) tablet.cc:699] Row locks acquired
1028 14:52:16.874586 (+ 3us) write_op.cc:260] PREPARE: finished
1028 14:52:16.874644 (+ 58us) op_driver.cc:389] HandleFailure(Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.001s ago.)
1028 14:52:16.874652 (+ 8us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.874663 (+ 11us) write_op.cc:454] Released schema lock
1028 14:52:16.874667 (+ 4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":1192402}]]}
W20251028 14:52:16.876969 3814 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:16.877355 3814 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47518 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=62, attempt_no=9}) took 1202 ms. Trace:
I20251028 14:52:16.877465 3814 rpcz_store.cc:276] 1028 14:52:15.675339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.675392 (+ 53us) service_pool.cc:225] Handling call
1028 14:52:16.877286 (+1201894us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:16.876130 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:16.876183 (+ 53us) op_driver.cc:389] HandleFailure(Illegal state: Transaction 11 already open)
1028 14:52:16.876190 (+ 7us) participant_op.cc:94] Released txn lock
1028 14:52:16.876195 (+ 5us) participant_op.cc:287] FINISH: Op aborted
Related trace 'op':
1028 14:52:16.876867 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER)
1028 14:52:16.876880 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.876886 (+ 6us) write_op.cc:454] Released schema lock
1028 14:52:16.876896 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":49,"txn-participant-registration.run_cpu_time_us":591,"txn-participant-registration.run_wall_time_us":6626,"child_traces":[["op",{"prepare.queue_time_us":1193753,"child_traces":[["op",{}]]}]]}
W20251028 14:52:16.880189 3814 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: Cannot bump the last assigned timestamp. Tablet is not in leader mode. Last heard from a leader: 0.002s ago.
I20251028 14:52:16.880465 3814 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.1:45449 (ReqId={client: fb71aec3fc3c438fbf5a1ebc1bf1c937, seq_no=82, attempt_no=1}) took 1198 ms. Trace:
I20251028 14:52:16.880638 3814 rpcz_store.cc:276] 1028 14:52:15.682206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.682433 (+ 227us) service_pool.cc:225] Handling call
1028 14:52:16.880426 (+1197993us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:16.880003 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:16.880101 (+ 98us) op_driver.cc:389] HandleFailure(Illegal state: Cannot bump the last assigned timestamp. Tablet is not in leader mode. Last heard from a leader: 0.002s ago.)
1028 14:52:16.880110 (+ 9us) participant_op.cc:94] Released txn lock
1028 14:52:16.880115 (+ 5us) participant_op.cc:287] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":1197334}]]}
W20251028 14:52:16.881280 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:16.882790 3814 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
I20251028 14:52:16.883064 3814 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47518 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=78, attempt_no=0}) took 1188 ms. Trace:
I20251028 14:52:16.883139 3814 rpcz_store.cc:276] 1028 14:52:15.694885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.694945 (+ 60us) service_pool.cc:225] Handling call
1028 14:52:16.883004 (+1188059us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:16.882395 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:16.882448 (+ 53us) op_driver.cc:389] HandleFailure(Illegal state: Transaction 14 already open)
1028 14:52:16.882456 (+ 8us) participant_op.cc:94] Released txn lock
1028 14:52:16.882461 (+ 5us) participant_op.cc:287] FINISH: Op aborted
Related trace 'op':
1028 14:52:16.882731 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER)
1028 14:52:16.882741 (+ 10us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.882747 (+ 6us) write_op.cc:454] Released schema lock
1028 14:52:16.882752 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":61,"txn-participant-registration.run_cpu_time_us":713,"txn-participant-registration.run_wall_time_us":2412,"child_traces":[["op",{"prepare.queue_time_us":1184590,"child_traces":[["op",{}]]}]]}
I20251028 14:52:16.885669 3904 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:54246 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=69, attempt_no=8}) took 1326 ms. Trace:
I20251028 14:52:16.885846 3904 rpcz_store.cc:276] 1028 14:52:15.558918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.560011 (+ 1093us) service_pool.cc:225] Handling call
1028 14:52:16.885594 (+1325583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:15.590283 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:15.590360 (+ 77us) participant_op.cc:199] PREPARE: finished
1028 14:52:15.590393 (+ 33us) participant_op.cc:212] START. Timestamp: 1761663135590346
1028 14:52:15.590396 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:52:15.590539 (+ 143us) log.cc:844] Serialized 65 byte log entry
1028 14:52:15.607785 (+ 17246us) op_driver.cc:464] REPLICATION: finished
1028 14:52:15.608811 (+ 1026us) participant_op.cc:265] APPLY: starting
1028 14:52:15.609587 (+ 776us) participant_op.cc:270] APPLY: finished
1028 14:52:15.609622 (+ 35us) log.cc:844] Serialized 14 byte log entry
1028 14:52:15.609649 (+ 27us) participant_op.cc:94] Released txn lock
1028 14:52:15.609655 (+ 6us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:52:15.610043 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c69e5c1a69cf438180b9f398eeaba707
1028 14:52:15.616473 (+ 6430us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:15.616476 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:15.616478 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:15.616564 (+ 86us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:52:15.616623 (+ 59us) tablet.cc:699] Row locks acquired
1028 14:52:15.616625 (+ 2us) write_op.cc:260] PREPARE: finished
1028 14:52:15.616681 (+ 56us) write_op.cc:270] Start()
1028 14:52:15.616714 (+ 33us) write_op.cc:276] Timestamp: P: 1761663135616634 usec, L: 0
1028 14:52:15.616717 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:52:15.616951 (+ 234us) log.cc:844] Serialized 302 byte log entry
1028 14:52:16.876033 (+1259082us) op_driver.cc:464] REPLICATION: finished
1028 14:52:16.884902 (+ 8869us) write_op.cc:301] APPLY: starting
1028 14:52:16.884930 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
1028 14:52:16.885004 (+ 74us) tablet.cc:1368] finished BulkCheckPresence
1028 14:52:16.885007 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:52:16.885136 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:52:16.885208 (+ 72us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
1028 14:52:16.885216 (+ 8us) write_op.cc:312] APPLY: finished
1028 14:52:16.885360 (+ 144us) log.cc:844] Serialized 24 byte log entry
1028 14:52:16.885464 (+ 104us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.885485 (+ 21us) write_op.cc:454] Released schema lock
1028 14:52:16.885492 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"txn-participant-registration.queue_time_us":298,"txn-participant-registration.run_cpu_time_us":672,"txn-participant-registration.run_wall_time_us":27028,"child_traces":[["op",{"apply.queue_time_us":996,"apply.run_cpu_time_us":678,"apply.run_wall_time_us":1145,"prepare.queue_time_us":2770,"prepare.run_cpu_time_us":325,"prepare.run_wall_time_us":359,"replication_time_us":17372,"spinlock_wait_cycles":1280,"child_traces":[["op",{"apply.queue_time_us":8780,"mutex_wait_us":3025,"num_ops":1,"prepare.queue_time_us":115,"prepare.run_cpu_time_us":782,"prepare.run_wall_time_us":10015,"replication_time_us":1259257,"threads_started":1}]]}]]}
W20251028 14:52:16.889183 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:16.896992 3907 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:54246 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=70, attempt_no=7}) took 1337 ms. Trace:
I20251028 14:52:16.899708 3907 rpcz_store.cc:276] 1028 14:52:15.559027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.560256 (+ 1229us) service_pool.cc:225] Handling call
1028 14:52:16.896931 (+1336675us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:15.591288 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:15.591319 (+ 31us) participant_op.cc:199] PREPARE: finished
1028 14:52:15.591341 (+ 22us) participant_op.cc:212] START. Timestamp: 1761663135591300
1028 14:52:15.591343 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:52:15.591433 (+ 90us) log.cc:844] Serialized 65 byte log entry
1028 14:52:15.607876 (+ 16443us) op_driver.cc:464] REPLICATION: finished
1028 14:52:15.609214 (+ 1338us) participant_op.cc:265] APPLY: starting
1028 14:52:15.620647 (+ 11433us) participant_op.cc:270] APPLY: finished
1028 14:52:15.620669 (+ 22us) log.cc:844] Serialized 14 byte log entry
1028 14:52:15.620713 (+ 44us) participant_op.cc:94] Released txn lock
1028 14:52:15.620720 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:52:15.622040 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c69e5c1a69cf438180b9f398eeaba707
1028 14:52:15.622171 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:15.622174 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:15.622176 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:15.622259 (+ 83us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:52:15.622316 (+ 57us) tablet.cc:699] Row locks acquired
1028 14:52:15.622318 (+ 2us) write_op.cc:260] PREPARE: finished
1028 14:52:15.622364 (+ 46us) write_op.cc:270] Start()
1028 14:52:15.622393 (+ 29us) write_op.cc:276] Timestamp: P: 1761663135622320 usec, L: 0
1028 14:52:15.622397 (+ 4us) op_driver.cc:348] REPLICATION: starting
1028 14:52:15.623197 (+ 800us) log.cc:844] Serialized 302 byte log entry
1028 14:52:16.895442 (+1272245us) op_driver.cc:464] REPLICATION: finished
1028 14:52:16.896348 (+ 906us) write_op.cc:301] APPLY: starting
1028 14:52:16.896371 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
1028 14:52:16.896437 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
1028 14:52:16.896439 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:52:16.896551 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:52:16.896632 (+ 81us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
1028 14:52:16.896640 (+ 8us) write_op.cc:312] APPLY: finished
1028 14:52:16.896709 (+ 69us) log.cc:844] Serialized 24 byte log entry
1028 14:52:16.896801 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.896821 (+ 20us) write_op.cc:454] Released schema lock
1028 14:52:16.896827 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"txn-participant-registration.queue_time_us":881,"txn-participant-registration.run_cpu_time_us":580,"txn-participant-registration.run_wall_time_us":27929,"child_traces":[["op",{"apply.queue_time_us":1292,"apply.run_cpu_time_us":702,"apply.run_wall_time_us":12610,"prepare.queue_time_us":2033,"prepare.run_cpu_time_us":177,"prepare.run_wall_time_us":177,"replication_time_us":16518,"child_traces":[["op",{"apply.queue_time_us":851,"mutex_wait_us":853,"num_ops":1,"prepare.queue_time_us":1123,"prepare.run_cpu_time_us":592,"prepare.run_wall_time_us":1219,"replication_time_us":1273011,"threads_started":1}]]}]]}
W20251028 14:52:16.902273 3905 tablet_replica.cc:1306] Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 19 similar messages]
W20251028 14:52:16.902412 3905 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:16.903244 3909 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:16.904285 3906 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:16.905710 3560 raft_consensus.cc:1275] T c69e5c1a69cf438180b9f398eeaba707 P 27f3e1cdaf5f4d6c957b5376bcbde0e5 [term 57 FOLLOWER]: Refusing update from remote peer d400e7c613e146ab91729d201413b76c: Log matching property violated. Preceding OpId in replica: term: 51 index: 85. Preceding OpId from leader: term: 57 index: 98. (index mismatch)
W20251028 14:52:16.908636 3908 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:16.909098 3895 consensus_queue.cc:1048] T c69e5c1a69cf438180b9f398eeaba707 P d400e7c613e146ab91729d201413b76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27f3e1cdaf5f4d6c957b5376bcbde0e5" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 84, Time since last communication: 0.000s
I20251028 14:52:16.911039 3910 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:54246 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=63, attempt_no=8}) took 1352 ms. Trace:
I20251028 14:52:16.911196 3910 rpcz_store.cc:276] 1028 14:52:15.558824 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.559622 (+ 798us) service_pool.cc:225] Handling call
1028 14:52:16.910972 (+1351350us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:15.599379 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:15.599418 (+ 39us) participant_op.cc:199] PREPARE: finished
1028 14:52:15.599453 (+ 35us) participant_op.cc:212] START. Timestamp: 1761663135599406
1028 14:52:15.599456 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:52:15.599619 (+ 163us) log.cc:844] Serialized 65 byte log entry
1028 14:52:15.623335 (+ 23716us) op_driver.cc:464] REPLICATION: finished
1028 14:52:15.623655 (+ 320us) participant_op.cc:265] APPLY: starting
1028 14:52:15.641599 (+ 17944us) participant_op.cc:270] APPLY: finished
1028 14:52:15.641633 (+ 34us) log.cc:844] Serialized 14 byte log entry
1028 14:52:15.641668 (+ 35us) participant_op.cc:94] Released txn lock
1028 14:52:15.641675 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:52:15.647452 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c69e5c1a69cf438180b9f398eeaba707
1028 14:52:15.647585 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:15.647588 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:15.647591 (+ 3us) tablet.cc:660] Decoding operations
1028 14:52:15.647676 (+ 85us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:52:15.647735 (+ 59us) tablet.cc:699] Row locks acquired
1028 14:52:15.647737 (+ 2us) write_op.cc:260] PREPARE: finished
1028 14:52:15.647787 (+ 50us) write_op.cc:270] Start()
1028 14:52:15.647818 (+ 31us) write_op.cc:276] Timestamp: P: 1761663135647740 usec, L: 0
1028 14:52:15.647822 (+ 4us) op_driver.cc:348] REPLICATION: starting
1028 14:52:15.648568 (+ 746us) log.cc:844] Serialized 302 byte log entry
1028 14:52:16.903947 (+1255379us) op_driver.cc:464] REPLICATION: finished
1028 14:52:16.906269 (+ 2322us) write_op.cc:301] APPLY: starting
1028 14:52:16.906302 (+ 33us) tablet.cc:1365] starting BulkCheckPresence
1028 14:52:16.906390 (+ 88us) tablet.cc:1368] finished BulkCheckPresence
1028 14:52:16.906393 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:52:16.906555 (+ 162us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:52:16.906669 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
1028 14:52:16.906679 (+ 10us) write_op.cc:312] APPLY: finished
1028 14:52:16.910729 (+ 4050us) log.cc:844] Serialized 24 byte log entry
1028 14:52:16.910820 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.910836 (+ 16us) write_op.cc:454] Released schema lock
1028 14:52:16.910842 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"txn-participant-registration.queue_time_us":57,"txn-participant-registration.run_cpu_time_us":739,"txn-participant-registration.run_wall_time_us":38630,"child_traces":[["op",{"apply.queue_time_us":270,"apply.run_cpu_time_us":708,"apply.run_wall_time_us":22986,"prepare.queue_time_us":737,"prepare.run_cpu_time_us":295,"prepare.run_wall_time_us":296,"replication_time_us":23838,"child_traces":[["op",{"apply.queue_time_us":2230,"num_ops":1,"prepare.queue_time_us":832,"prepare.run_cpu_time_us":729,"prepare.run_wall_time_us":1212,"raft.queue_time_us":18759,"raft.run_cpu_time_us":482,"raft.run_wall_time_us":1736,"replication_time_us":1256089,"spinlock_wait_cycles":1259904}]]}]]}
I20251028 14:52:16.927807 3904 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:54246 (ReqId={client: f65174607ff54be2b76cb4e58afd9912, seq_no=74, attempt_no=0}) took 1329 ms. Trace:
I20251028 14:52:16.928004 3904 rpcz_store.cc:276] 1028 14:52:15.598653 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:15.598705 (+ 52us) service_pool.cc:225] Handling call
1028 14:52:16.927726 (+1329021us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:15.621477 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:15.621513 (+ 36us) participant_op.cc:199] PREPARE: finished
1028 14:52:15.621555 (+ 42us) participant_op.cc:212] START. Timestamp: 1761663135621497
1028 14:52:15.621558 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:52:15.621947 (+ 389us) log.cc:844] Serialized 65 byte log entry
1028 14:52:16.887904 (+1265957us) op_driver.cc:464] REPLICATION: finished
1028 14:52:16.895531 (+ 7627us) participant_op.cc:265] APPLY: starting
1028 14:52:16.895898 (+ 367us) participant_op.cc:270] APPLY: finished
1028 14:52:16.895949 (+ 51us) log.cc:844] Serialized 14 byte log entry
1028 14:52:16.895988 (+ 39us) participant_op.cc:94] Released txn lock
1028 14:52:16.895995 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:52:16.905107 (+ 0us) write_op.cc:183] PREPARE: starting on tablet c69e5c1a69cf438180b9f398eeaba707
1028 14:52:16.910412 (+ 5305us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:16.910416 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:52:16.910418 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:16.910519 (+ 101us) tablet.cc:683] Acquiring locks for 1 operations
1028 14:52:16.910577 (+ 58us) tablet.cc:699] Row locks acquired
1028 14:52:16.910579 (+ 2us) write_op.cc:260] PREPARE: finished
1028 14:52:16.910630 (+ 51us) write_op.cc:270] Start()
1028 14:52:16.910676 (+ 46us) write_op.cc:276] Timestamp: P: 1761663136910612 usec, L: 0
1028 14:52:16.910679 (+ 3us) op_driver.cc:348] REPLICATION: starting
1028 14:52:16.910913 (+ 234us) log.cc:844] Serialized 302 byte log entry
1028 14:52:16.925533 (+ 14620us) op_driver.cc:464] REPLICATION: finished
1028 14:52:16.926526 (+ 993us) write_op.cc:301] APPLY: starting
1028 14:52:16.926662 (+ 136us) tablet.cc:1365] starting BulkCheckPresence
1028 14:52:16.927274 (+ 612us) tablet.cc:1368] finished BulkCheckPresence
1028 14:52:16.927277 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
1028 14:52:16.927391 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
1028 14:52:16.927446 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
1028 14:52:16.927455 (+ 9us) write_op.cc:312] APPLY: finished
1028 14:52:16.927522 (+ 67us) log.cc:844] Serialized 24 byte log entry
1028 14:52:16.927585 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:16.927603 (+ 18us) write_op.cc:454] Released schema lock
1028 14:52:16.927611 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"txn-participant-registration.queue_time_us":675,"txn-participant-registration.run_cpu_time_us":643,"txn-participant-registration.run_wall_time_us":21981,"child_traces":[["op",{"apply.queue_time_us":7536,"apply.run_cpu_time_us":753,"apply.run_wall_time_us":753,"prepare.queue_time_us":610,"prepare.run_cpu_time_us":382,"prepare.run_wall_time_us":521,"replication_time_us":1266317,"spinlock_wait_cycles":543232,"thread_start_us":7376,"threads_started":1,"child_traces":[["op",{"apply.queue_time_us":933,"num_ops":1,"prepare.queue_time_us":8847,"prepare.run_cpu_time_us":775,"prepare.run_wall_time_us":5884,"replication_time_us":14819}]]}]]}
W20251028 14:52:16.952391 3381 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:16.955410 3382 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:16.959689 3515 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:16.964326 3908 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:49828: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
W20251028 14:52:16.989476 3403 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.ParticipateInTransaction -> {remote=127.0.104.3:37725, user_credentials={real_user=slave}} blocked reactor thread for 51825us
W20251028 14:52:16.990635 3912 tablet_replica.cc:1306] Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 18 similar messages]
W20251028 14:52:16.990751 3912 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:17.001475 3894 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations [suppressed 9 similar messages]
W20251028 14:52:17.006729 3402 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:45449: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:17.017516 3924 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:17.019050 3911 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:17.020174 3923 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:17.017560 3921 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:17.020716 3914 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47518: Illegal state: replica d7cc28c72b6e454f98cc0096e3c57a1d is not leader of this config: current role FOLLOWER
W20251028 14:52:17.025627 3544 tablet_service.cc:731] failed op from {username='slave'} at 127.0.104.1:39779: Illegal state: replica 27f3e1cdaf5f4d6c957b5376bcbde0e5 is not leader of this config: current role FOLLOWER
I20251028 14:52:17.301096 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 3338
I20251028 14:52:17.303032 3468 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:17.468091 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 3338
W20251028 14:52:17.505931 3248 connection.cc:537] server connection from 127.0.104.1:44739 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:52:17.506420 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 3475
I20251028 14:52:17.507653 3609 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:17.644495 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 3475
W20251028 14:52:17.673771 3631 connection.cc:537] client connection to 127.0.104.2:41917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 14:52:17.674095 3631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 14:52:17.674216 3250 connection.cc:537] server connection from 127.0.104.2:45691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:52:17.674517 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 3616
I20251028 14:52:17.675803 3745 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:17.849972 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 3616
I20251028 14:52:17.873236 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 3233
W20251028 14:52:17.873186 3251 connection.cc:537] server connection from 127.0.104.3:51457 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:52:17.874641 3304 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:17.997458 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 3233
2025-10-28T14:52:18Z chronyd exiting
[ OK ] TxnWriteOpsITest.FrequentElections (7388 ms)
[ RUN ] TxnWriteOpsITest.WriteOpPerf
2025-10-28T14:52:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:52:18Z Disabled control of system clock
I20251028 14:52:18.066965 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35765
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35765
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20251028 14:52:18.418072 3954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:18.418402 3954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:18.418493 3954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:18.427029 3954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:52:18.427143 3954 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:52:18.427191 3954 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20251028 14:52:18.427227 3954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:18.427307 3954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:52:18.427368 3954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:52:18.437832 3954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35765
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:18.439630 3954 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:18.441608 3954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:18.450402 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:18.450335 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:18.451848 3954 server_base.cc:1047] running on GCE node
W20251028 14:52:18.452051 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:18.452915 3954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:18.454725 3954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:18.456132 3954 hybrid_clock.cc:648] HybridClock initialized: now 1761663138456069 us; error 69 us; skew 500 ppm
I20251028 14:52:18.456630 3954 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:18.459445 3954 webserver.cc:492] Webserver started at http://127.0.104.62:34263/ using document root <none> and password file <none>
I20251028 14:52:18.460296 3954 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:18.460407 3954 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:18.460815 3954 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:18.465019 3954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "900d21d23e964445bcc994529f99e5fd"
format_stamp: "Formatted at 2025-10-28 14:52:18 on dist-test-slave-0v9m"
I20251028 14:52:18.466041 3954 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "900d21d23e964445bcc994529f99e5fd"
format_stamp: "Formatted at 2025-10-28 14:52:18 on dist-test-slave-0v9m"
I20251028 14:52:18.472325 3954 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251028 14:52:18.476994 3968 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:18.478652 3954 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:18.478863 3954 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "900d21d23e964445bcc994529f99e5fd"
format_stamp: "Formatted at 2025-10-28 14:52:18 on dist-test-slave-0v9m"
I20251028 14:52:18.479140 3954 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:18.518349 3954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:18.519485 3954 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:18.519807 3954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:18.546101 3954 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35765
I20251028 14:52:18.546106 4029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35765 every 8 connection(s)
I20251028 14:52:18.547755 3954 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:52:18.549144 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 3954
I20251028 14:52:18.549371 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:52:18.553858 4030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:18.565443 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd: Bootstrap starting.
I20251028 14:52:18.569038 4030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:18.570353 4030 log.cc:826] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:18.573659 4030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd: No bootstrap required, opened a new log
I20251028 14:52:18.579785 4030 raft_consensus.cc:359] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900d21d23e964445bcc994529f99e5fd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35765 } }
I20251028 14:52:18.580214 4030 raft_consensus.cc:385] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:18.580314 4030 raft_consensus.cc:740] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 900d21d23e964445bcc994529f99e5fd, State: Initialized, Role: FOLLOWER
I20251028 14:52:18.580960 4030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900d21d23e964445bcc994529f99e5fd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35765 } }
I20251028 14:52:18.581176 4030 raft_consensus.cc:399] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:18.581326 4030 raft_consensus.cc:493] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:18.581486 4030 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:18.583940 4030 raft_consensus.cc:515] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900d21d23e964445bcc994529f99e5fd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35765 } }
I20251028 14:52:18.584679 4030 leader_election.cc:304] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 900d21d23e964445bcc994529f99e5fd; no voters:
I20251028 14:52:18.585417 4030 leader_election.cc:290] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:18.585637 4035 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:18.586308 4035 raft_consensus.cc:697] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [term 1 LEADER]: Becoming Leader. State: Replica: 900d21d23e964445bcc994529f99e5fd, State: Running, Role: LEADER
I20251028 14:52:18.586846 4035 consensus_queue.cc:237] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900d21d23e964445bcc994529f99e5fd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35765 } }
I20251028 14:52:18.587543 4030 sys_catalog.cc:565] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:18.589742 4037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 900d21d23e964445bcc994529f99e5fd. Latest consensus state: current_term: 1 leader_uuid: "900d21d23e964445bcc994529f99e5fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900d21d23e964445bcc994529f99e5fd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35765 } } }
I20251028 14:52:18.590046 4037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:18.591008 4041 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:18.591030 4036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "900d21d23e964445bcc994529f99e5fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "900d21d23e964445bcc994529f99e5fd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35765 } } }
I20251028 14:52:18.591320 4036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:18.597164 4041 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:18.611099 4041 catalog_manager.cc:1357] Generated new cluster ID: 5c9f16ea3b7a4a429eae15dc48ff4685
I20251028 14:52:18.611253 4041 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:18.641348 4041 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:18.643283 4041 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:18.669610 4041 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 900d21d23e964445bcc994529f99e5fd: Generated new TSK 0
I20251028 14:52:18.670639 4041 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:18.686605 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35765
--builtin_ntp_servers=127.0.104.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tablet_max_pending_txn_write_ops=10
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20251028 14:52:18.691413 3985 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33932:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:18.701772 4030 master.cc:464] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20251028 14:52:19.062134 4059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:19.062621 4059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:19.062747 4059 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20251028 14:52:19.062881 4059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:19.072988 4059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:19.073225 4059 flags.cc:425] Enabled experimental flag: --tablet_max_pending_txn_write_ops=10
W20251028 14:52:19.073334 4059 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:19.073457 4059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:52:19.085374 4059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_partition_lock=false
--tablet_max_pending_txn_write_ops=10
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:19.087846 4059 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:19.090667 4059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:19.102725 4064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:19.102696 4065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:19.103665 4067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:19.104947 4059 server_base.cc:1047] running on GCE node
I20251028 14:52:19.105710 4059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:19.106911 4059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:19.108192 4059 hybrid_clock.cc:648] HybridClock initialized: now 1761663139108115 us; error 46 us; skew 500 ppm
I20251028 14:52:19.108563 4059 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:19.111426 4059 webserver.cc:492] Webserver started at http://127.0.104.1:33751/ using document root <none> and password file <none>
I20251028 14:52:19.112258 4059 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:19.112398 4059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:19.112785 4059 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:19.115531 4059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "b4691314ae524e4aaa16630b81a79cfd"
format_stamp: "Formatted at 2025-10-28 14:52:19 on dist-test-slave-0v9m"
I20251028 14:52:19.116429 4059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "b4691314ae524e4aaa16630b81a79cfd"
format_stamp: "Formatted at 2025-10-28 14:52:19 on dist-test-slave-0v9m"
I20251028 14:52:19.123431 4059 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20251028 14:52:19.127743 4073 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:19.129662 4059 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:19.130028 4059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "b4691314ae524e4aaa16630b81a79cfd"
format_stamp: "Formatted at 2025-10-28 14:52:19 on dist-test-slave-0v9m"
I20251028 14:52:19.130265 4059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:19.152523 4059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:19.153565 4059 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:19.153867 4059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:19.157325 4059 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:19.157505 4059 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251028 14:52:19.157671 4059 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:19.157776 4059 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:19.209288 4059 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39721
I20251028 14:52:19.209395 4193 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39721 every 8 connection(s)
I20251028 14:52:19.210855 4059 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:52:19.211980 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 4059
I20251028 14:52:19.212234 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:52:19.214007 4194 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35765
I20251028 14:52:19.214334 4194 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:19.215263 4194 heartbeater.cc:507] Master 127.0.104.62:35765 requested a full tablet report, sending...
I20251028 14:52:19.215643 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35765
--builtin_ntp_servers=127.0.104.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tablet_max_pending_txn_write_ops=10
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20251028 14:52:19.217515 3985 ts_manager.cc:194] Registered new tserver with Master: b4691314ae524e4aaa16630b81a79cfd (127.0.104.1:39721)
I20251028 14:52:19.219177 3985 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:34173
W20251028 14:52:19.602133 4196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:19.602459 4196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:19.602542 4196 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20251028 14:52:19.602630 4196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:19.612438 4196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:19.612651 4196 flags.cc:425] Enabled experimental flag: --tablet_max_pending_txn_write_ops=10
W20251028 14:52:19.612706 4196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:19.612797 4196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:52:19.623646 4196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_partition_lock=false
--tablet_max_pending_txn_write_ops=10
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:19.625406 4196 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:19.627385 4196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:19.638279 4202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:19.638305 4201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:19.638973 4204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:19.639242 4196 server_base.cc:1047] running on GCE node
I20251028 14:52:19.639879 4196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:19.640962 4196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:19.642230 4196 hybrid_clock.cc:648] HybridClock initialized: now 1761663139642136 us; error 72 us; skew 500 ppm
I20251028 14:52:19.642571 4196 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:19.645447 4196 webserver.cc:492] Webserver started at http://127.0.104.2:45951/ using document root <none> and password file <none>
I20251028 14:52:19.646265 4196 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:19.646399 4196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:19.646750 4196 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:19.649303 4196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2"
format_stamp: "Formatted at 2025-10-28 14:52:19 on dist-test-slave-0v9m"
I20251028 14:52:19.650108 4196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2"
format_stamp: "Formatted at 2025-10-28 14:52:19 on dist-test-slave-0v9m"
I20251028 14:52:19.655979 4196 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.005s
I20251028 14:52:19.660492 4210 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:19.662271 4196 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20251028 14:52:19.662472 4196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2"
format_stamp: "Formatted at 2025-10-28 14:52:19 on dist-test-slave-0v9m"
I20251028 14:52:19.662689 4196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:19.694867 4196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:19.695799 4196 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:19.696086 4196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:19.699383 4196 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:19.699496 4196 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:19.699628 4196 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:19.699710 4196 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:19.744259 3985 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33938:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:19.759569 4196 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43431
I20251028 14:52:19.759678 4336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43431 every 8 connection(s)
I20251028 14:52:19.761194 4196 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:52:19.763938 4337 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35765
I20251028 14:52:19.764256 4337 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:19.765154 4337 heartbeater.cc:507] Master 127.0.104.62:35765 requested a full tablet report, sending...
I20251028 14:52:19.766923 3985 ts_manager.cc:194] Registered new tserver with Master: b5dcbffb7bfd4af29b6e248ba2d09ac2 (127.0.104.2:43431)
I20251028 14:52:19.767750 3985 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:39319
I20251028 14:52:19.770316 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 4196
I20251028 14:52:19.770505 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:52:19.773983 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35765
--builtin_ntp_servers=127.0.104.20:44823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tablet_max_pending_txn_write_ops=10
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
W20251028 14:52:20.124078 4339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:20.124545 4339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:20.124677 4339 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20251028 14:52:20.124801 4339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:20.135967 4339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:20.136158 4339 flags.cc:425] Enabled experimental flag: --tablet_max_pending_txn_write_ops=10
W20251028 14:52:20.136210 4339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:20.136317 4339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:52:20.146890 4339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_partition_lock=false
--tablet_max_pending_txn_write_ops=10
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35765
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:20.148540 4339 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:20.150516 4339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:20.161088 4345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:20.161211 4344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:20.161440 4347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:20.162711 4339 server_base.cc:1047] running on GCE node
I20251028 14:52:20.163347 4339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:20.164408 4339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:20.165645 4339 hybrid_clock.cc:648] HybridClock initialized: now 1761663140165577 us; error 37 us; skew 500 ppm
I20251028 14:52:20.165999 4339 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:20.169600 4339 webserver.cc:492] Webserver started at http://127.0.104.3:33127/ using document root <none> and password file <none>
I20251028 14:52:20.170473 4339 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:20.170598 4339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:20.170977 4339 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:20.174162 4339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "52dc96db7be7426f8dc151f7ef365234"
format_stamp: "Formatted at 2025-10-28 14:52:20 on dist-test-slave-0v9m"
I20251028 14:52:20.175231 4339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "52dc96db7be7426f8dc151f7ef365234"
format_stamp: "Formatted at 2025-10-28 14:52:20 on dist-test-slave-0v9m"
I20251028 14:52:20.181685 4339 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20251028 14:52:20.186028 4353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:20.188352 4339 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20251028 14:52:20.188665 4339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "52dc96db7be7426f8dc151f7ef365234"
format_stamp: "Formatted at 2025-10-28 14:52:20 on dist-test-slave-0v9m"
I20251028 14:52:20.189136 4339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:20.212366 4339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:20.213639 4339 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:20.214015 4339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:20.217571 4339 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:20.217726 4339 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:20.217854 4339 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:20.217943 4339 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:20.222375 4194 heartbeater.cc:499] Master 127.0.104.62:35765 was elected leader, sending a full tablet report...
I20251028 14:52:20.263782 4339 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:40259
I20251028 14:52:20.263867 4472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:40259 every 8 connection(s)
I20251028 14:52:20.265506 4339 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:52:20.266247 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 4339
I20251028 14:52:20.266446 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:52:20.268203 4473 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35765
I20251028 14:52:20.268605 4473 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:20.269500 4473 heartbeater.cc:507] Master 127.0.104.62:35765 requested a full tablet report, sending...
I20251028 14:52:20.271366 3984 ts_manager.cc:194] Registered new tserver with Master: 52dc96db7be7426f8dc151f7ef365234 (127.0.104.3:40259)
I20251028 14:52:20.272279 3984 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:47509
I20251028 14:52:20.283291 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:52:20.303287 3984 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33946:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:52:20.304370 3984 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:20.337481 4129 tablet_service.cc:1505] Processing CreateTablet for tablet 95e350922bf34f32ae3506cf73130aef (DEFAULT_TABLE table=txn_write_ops_test [id=cf18d3d5dc574f77b3990dfe0bc40d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:20.338127 4265 tablet_service.cc:1505] Processing CreateTablet for tablet 03eec246e7d1403ba7d079d536d18f31 (DEFAULT_TABLE table=txn_write_ops_test [id=cf18d3d5dc574f77b3990dfe0bc40d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:20.339190 4129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e350922bf34f32ae3506cf73130aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.339354 4128 tablet_service.cc:1505] Processing CreateTablet for tablet 03eec246e7d1403ba7d079d536d18f31 (DEFAULT_TABLE table=txn_write_ops_test [id=cf18d3d5dc574f77b3990dfe0bc40d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:20.339546 4265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eec246e7d1403ba7d079d536d18f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.339901 4128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eec246e7d1403ba7d079d536d18f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.342304 4408 tablet_service.cc:1505] Processing CreateTablet for tablet 95e350922bf34f32ae3506cf73130aef (DEFAULT_TABLE table=txn_write_ops_test [id=cf18d3d5dc574f77b3990dfe0bc40d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:20.344471 4408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e350922bf34f32ae3506cf73130aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.348482 4407 tablet_service.cc:1505] Processing CreateTablet for tablet 03eec246e7d1403ba7d079d536d18f31 (DEFAULT_TABLE table=txn_write_ops_test [id=cf18d3d5dc574f77b3990dfe0bc40d56]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:20.349138 4407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03eec246e7d1403ba7d079d536d18f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.353376 4266 tablet_service.cc:1505] Processing CreateTablet for tablet 95e350922bf34f32ae3506cf73130aef (DEFAULT_TABLE table=txn_write_ops_test [id=cf18d3d5dc574f77b3990dfe0bc40d56]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:20.353960 4266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95e350922bf34f32ae3506cf73130aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.359938 4491 tablet_bootstrap.cc:492] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd: Bootstrap starting.
I20251028 14:52:20.363379 4491 tablet_bootstrap.cc:654] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.364543 4490 tablet_bootstrap.cc:492] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Bootstrap starting.
I20251028 14:52:20.364759 4492 tablet_bootstrap.cc:492] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234: Bootstrap starting.
I20251028 14:52:20.364868 4491 log.cc:826] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:20.368937 4491 tablet_bootstrap.cc:492] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd: No bootstrap required, opened a new log
I20251028 14:52:20.369320 4491 ts_tablet_manager.cc:1403] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20251028 14:52:20.369652 4490 tablet_bootstrap.cc:654] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.369637 4492 tablet_bootstrap.cc:654] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.371093 4492 log.cc:826] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:20.371093 4490 log.cc:826] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:20.375932 4491 raft_consensus.cc:359] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.376199 4490 tablet_bootstrap.cc:492] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2: No bootstrap required, opened a new log
I20251028 14:52:20.376199 4492 tablet_bootstrap.cc:492] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234: No bootstrap required, opened a new log
I20251028 14:52:20.376302 4491 raft_consensus.cc:385] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.376398 4491 raft_consensus.cc:740] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4691314ae524e4aaa16630b81a79cfd, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.376528 4490 ts_tablet_manager.cc:1403] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20251028 14:52:20.376525 4492 ts_tablet_manager.cc:1403] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20251028 14:52:20.377071 4491 consensus_queue.cc:260] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.378816 4491 ts_tablet_manager.cc:1434] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20251028 14:52:20.379312 4491 tablet_bootstrap.cc:492] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd: Bootstrap starting.
I20251028 14:52:20.381093 4491 tablet_bootstrap.cc:654] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.384279 4492 raft_consensus.cc:359] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.384279 4490 raft_consensus.cc:359] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.384779 4492 raft_consensus.cc:385] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.384779 4490 raft_consensus.cc:385] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.384893 4490 raft_consensus.cc:740] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dcbffb7bfd4af29b6e248ba2d09ac2, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.384903 4492 raft_consensus.cc:740] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc96db7be7426f8dc151f7ef365234, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.385701 4490 consensus_queue.cc:260] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.385699 4492 consensus_queue.cc:260] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.390340 4491 tablet_bootstrap.cc:492] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd: No bootstrap required, opened a new log
I20251028 14:52:20.390513 4491 ts_tablet_manager.cc:1403] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20251028 14:52:20.391194 4491 raft_consensus.cc:359] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.391347 4491 raft_consensus.cc:385] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.391428 4491 raft_consensus.cc:740] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4691314ae524e4aaa16630b81a79cfd, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.391686 4491 consensus_queue.cc:260] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.392185 4491 ts_tablet_manager.cc:1434] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 14:52:20.393054 4473 heartbeater.cc:499] Master 127.0.104.62:35765 was elected leader, sending a full tablet report...
I20251028 14:52:20.394454 4337 heartbeater.cc:499] Master 127.0.104.62:35765 was elected leader, sending a full tablet report...
I20251028 14:52:20.395206 4490 ts_tablet_manager.cc:1434] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20251028 14:52:20.396165 4492 ts_tablet_manager.cc:1434] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234: Time spent starting tablet: real 0.020s user 0.013s sys 0.006s
I20251028 14:52:20.396481 4490 tablet_bootstrap.cc:492] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2: Bootstrap starting.
I20251028 14:52:20.396571 4492 tablet_bootstrap.cc:492] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234: Bootstrap starting.
I20251028 14:52:20.398471 4490 tablet_bootstrap.cc:654] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.398468 4492 tablet_bootstrap.cc:654] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.403303 4490 tablet_bootstrap.cc:492] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2: No bootstrap required, opened a new log
I20251028 14:52:20.403548 4492 tablet_bootstrap.cc:492] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234: No bootstrap required, opened a new log
I20251028 14:52:20.403577 4490 ts_tablet_manager.cc:1403] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20251028 14:52:20.403734 4492 ts_tablet_manager.cc:1403] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20251028 14:52:20.404850 4492 raft_consensus.cc:359] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.404778 4490 raft_consensus.cc:359] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.405057 4490 raft_consensus.cc:385] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.405057 4492 raft_consensus.cc:385] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.405146 4490 raft_consensus.cc:740] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dcbffb7bfd4af29b6e248ba2d09ac2, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.405143 4492 raft_consensus.cc:740] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc96db7be7426f8dc151f7ef365234, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.405449 4490 consensus_queue.cc:260] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.405449 4492 consensus_queue.cc:260] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.406039 4492 ts_tablet_manager.cc:1434] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:20.406147 4490 ts_tablet_manager.cc:1434] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:20.450084 4499 raft_consensus.cc:493] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:20.450357 4499 raft_consensus.cc:515] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.452219 4499 leader_election.cc:290] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4691314ae524e4aaa16630b81a79cfd (127.0.104.1:39721), 52dc96db7be7426f8dc151f7ef365234 (127.0.104.3:40259)
I20251028 14:52:20.453123 4499 raft_consensus.cc:493] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:20.453465 4499 raft_consensus.cc:515] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.454321 4499 leader_election.cc:290] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4691314ae524e4aaa16630b81a79cfd (127.0.104.1:39721), 52dc96db7be7426f8dc151f7ef365234 (127.0.104.3:40259)
I20251028 14:52:20.461813 4428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03eec246e7d1403ba7d079d536d18f31" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc96db7be7426f8dc151f7ef365234" is_pre_election: true
I20251028 14:52:20.461768 4427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95e350922bf34f32ae3506cf73130aef" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc96db7be7426f8dc151f7ef365234" is_pre_election: true
I20251028 14:52:20.462360 4148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95e350922bf34f32ae3506cf73130aef" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4691314ae524e4aaa16630b81a79cfd" is_pre_election: true
I20251028 14:52:20.462591 4427 raft_consensus.cc:2468] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 0.
I20251028 14:52:20.462597 4428 raft_consensus.cc:2468] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 0.
I20251028 14:52:20.462599 4149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03eec246e7d1403ba7d079d536d18f31" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4691314ae524e4aaa16630b81a79cfd" is_pre_election: true
I20251028 14:52:20.462939 4148 raft_consensus.cc:2468] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 0.
I20251028 14:52:20.463032 4149 raft_consensus.cc:2468] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 0.
I20251028 14:52:20.463454 4212 leader_election.cc:304] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc96db7be7426f8dc151f7ef365234, b5dcbffb7bfd4af29b6e248ba2d09ac2; no voters:
W20251028 14:52:20.463819 4195 tablet.cc:2378] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:52:20.463954 4211 leader_election.cc:304] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4691314ae524e4aaa16630b81a79cfd, b5dcbffb7bfd4af29b6e248ba2d09ac2; no voters:
I20251028 14:52:20.464349 4499 raft_consensus.cc:2804] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:20.464546 4499 raft_consensus.cc:493] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:20.464701 4499 raft_consensus.cc:3060] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:20.464730 4504 raft_consensus.cc:2804] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:20.464824 4504 raft_consensus.cc:493] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:20.464955 4504 raft_consensus.cc:3060] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:20.467253 4499 raft_consensus.cc:515] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.467335 4504 raft_consensus.cc:515] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.468016 4499 leader_election.cc:290] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 election: Requested vote from peers b4691314ae524e4aaa16630b81a79cfd (127.0.104.1:39721), 52dc96db7be7426f8dc151f7ef365234 (127.0.104.3:40259)
I20251028 14:52:20.468079 4504 leader_election.cc:290] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 election: Requested vote from peers b4691314ae524e4aaa16630b81a79cfd (127.0.104.1:39721), 52dc96db7be7426f8dc151f7ef365234 (127.0.104.3:40259)
I20251028 14:52:20.468675 4149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03eec246e7d1403ba7d079d536d18f31" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4691314ae524e4aaa16630b81a79cfd"
I20251028 14:52:20.468686 4428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "03eec246e7d1403ba7d079d536d18f31" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc96db7be7426f8dc151f7ef365234"
I20251028 14:52:20.468755 4148 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95e350922bf34f32ae3506cf73130aef" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4691314ae524e4aaa16630b81a79cfd"
I20251028 14:52:20.468921 4428 raft_consensus.cc:3060] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:20.468933 4148 raft_consensus.cc:3060] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:20.468947 4149 raft_consensus.cc:3060] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:20.469148 4427 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95e350922bf34f32ae3506cf73130aef" candidate_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc96db7be7426f8dc151f7ef365234"
I20251028 14:52:20.469367 4427 raft_consensus.cc:3060] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:20.471777 4427 raft_consensus.cc:2468] T 95e350922bf34f32ae3506cf73130aef P 52dc96db7be7426f8dc151f7ef365234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 1.
I20251028 14:52:20.472432 4428 raft_consensus.cc:2468] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 1.
I20251028 14:52:20.472450 4148 raft_consensus.cc:2468] T 95e350922bf34f32ae3506cf73130aef P b4691314ae524e4aaa16630b81a79cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 1.
I20251028 14:52:20.472553 4212 leader_election.cc:304] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc96db7be7426f8dc151f7ef365234, b5dcbffb7bfd4af29b6e248ba2d09ac2; no voters:
I20251028 14:52:20.472934 4504 raft_consensus.cc:2804] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:20.473138 4504 raft_consensus.cc:697] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 1 LEADER]: Becoming Leader. State: Replica: b5dcbffb7bfd4af29b6e248ba2d09ac2, State: Running, Role: LEADER
I20251028 14:52:20.473171 4212 leader_election.cc:304] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc96db7be7426f8dc151f7ef365234, b5dcbffb7bfd4af29b6e248ba2d09ac2; no voters:
I20251028 14:52:20.473546 4499 raft_consensus.cc:2804] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:20.473698 4499 raft_consensus.cc:697] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 1 LEADER]: Becoming Leader. State: Replica: b5dcbffb7bfd4af29b6e248ba2d09ac2, State: Running, Role: LEADER
I20251028 14:52:20.473788 4504 consensus_queue.cc:237] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.473918 4499 consensus_queue.cc:237] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.475646 4149 raft_consensus.cc:2468] T 03eec246e7d1403ba7d079d536d18f31 P b4691314ae524e4aaa16630b81a79cfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5dcbffb7bfd4af29b6e248ba2d09ac2 in term 1.
I20251028 14:52:20.481058 3984 catalog_manager.cc:5649] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5dcbffb7bfd4af29b6e248ba2d09ac2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:20.488406 3984 catalog_manager.cc:5649] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5dcbffb7bfd4af29b6e248ba2d09ac2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:20.513582 4338 tablet.cc:2378] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:20.518350 4474 tablet.cc:2378] T 03eec246e7d1403ba7d079d536d18f31 P 52dc96db7be7426f8dc151f7ef365234: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:20.648916 4512 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:52:20.762522 3984 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34014:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:20.763487 3984 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:20.773586 4266 tablet_service.cc:1505] Processing CreateTablet for tablet 3202a0eb447744999af8a7b7d3618b08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c6d591fc6c24e408c7b8a26565a8a69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:20.773511 4128 tablet_service.cc:1505] Processing CreateTablet for tablet 3202a0eb447744999af8a7b7d3618b08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c6d591fc6c24e408c7b8a26565a8a69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:20.774574 4266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3202a0eb447744999af8a7b7d3618b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.774091 4407 tablet_service.cc:1505] Processing CreateTablet for tablet 3202a0eb447744999af8a7b7d3618b08 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7c6d591fc6c24e408c7b8a26565a8a69]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:20.774592 4128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3202a0eb447744999af8a7b7d3618b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.774940 4407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3202a0eb447744999af8a7b7d3618b08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:20.780289 4490 tablet_bootstrap.cc:492] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Bootstrap starting.
I20251028 14:52:20.782284 4490 tablet_bootstrap.cc:654] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.782873 4492 tablet_bootstrap.cc:492] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: Bootstrap starting.
I20251028 14:52:20.784438 4492 tablet_bootstrap.cc:654] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.785828 4491 tablet_bootstrap.cc:492] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: Bootstrap starting.
I20251028 14:52:20.786522 4492 tablet_bootstrap.cc:492] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: No bootstrap required, opened a new log
I20251028 14:52:20.786729 4492 ts_tablet_manager.cc:1403] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20251028 14:52:20.787595 4492 raft_consensus.cc:359] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.787756 4491 tablet_bootstrap.cc:654] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:20.787808 4492 raft_consensus.cc:385] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.787873 4492 raft_consensus.cc:740] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dc96db7be7426f8dc151f7ef365234, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.788478 4492 consensus_queue.cc:260] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.789151 4498 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:20.789448 4498 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:20.790066 4492 ts_tablet_manager.cc:1434] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251028 14:52:20.793284 4491 tablet_bootstrap.cc:492] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: No bootstrap required, opened a new log
I20251028 14:52:20.793509 4491 ts_tablet_manager.cc:1403] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20251028 14:52:20.794315 4490 tablet_bootstrap.cc:492] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: No bootstrap required, opened a new log
I20251028 14:52:20.794436 4491 raft_consensus.cc:359] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.794588 4490 ts_tablet_manager.cc:1403] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20251028 14:52:20.794675 4491 raft_consensus.cc:385] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.794765 4491 raft_consensus.cc:740] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4691314ae524e4aaa16630b81a79cfd, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.795058 4491 consensus_queue.cc:260] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.795393 4490 raft_consensus.cc:359] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.795601 4490 raft_consensus.cc:385] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:20.795675 4490 raft_consensus.cc:740] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5dcbffb7bfd4af29b6e248ba2d09ac2, State: Initialized, Role: FOLLOWER
I20251028 14:52:20.795526 4496 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:20.795856 4496 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:20.796304 4491 ts_tablet_manager.cc:1434] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251028 14:52:20.796649 4490 consensus_queue.cc:260] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:20.797156 4490 ts_tablet_manager.cc:1434] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251028 14:52:20.797163 4504 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:20.797500 4504 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:20.904600 4504 consensus_queue.cc:1048] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:20.913314 4504 consensus_queue.cc:1048] T 95e350922bf34f32ae3506cf73130aef P b5dcbffb7bfd4af29b6e248ba2d09ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:20.977908 4504 consensus_queue.cc:1048] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:20.982648 4499 consensus_queue.cc:1048] T 03eec246e7d1403ba7d079d536d18f31 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:21.074998 4496 raft_consensus.cc:493] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:21.075237 4496 raft_consensus.cc:515] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:21.076869 4496 leader_election.cc:290] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5dcbffb7bfd4af29b6e248ba2d09ac2 (127.0.104.2:43431), 52dc96db7be7426f8dc151f7ef365234 (127.0.104.3:40259)
I20251028 14:52:21.095625 4428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3202a0eb447744999af8a7b7d3618b08" candidate_uuid: "b4691314ae524e4aaa16630b81a79cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc96db7be7426f8dc151f7ef365234" is_pre_election: true
I20251028 14:52:21.095719 4292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3202a0eb447744999af8a7b7d3618b08" candidate_uuid: "b4691314ae524e4aaa16630b81a79cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" is_pre_election: true
I20251028 14:52:21.095892 4428 raft_consensus.cc:2468] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4691314ae524e4aaa16630b81a79cfd in term 0.
I20251028 14:52:21.096262 4292 raft_consensus.cc:2468] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4691314ae524e4aaa16630b81a79cfd in term 0.
I20251028 14:52:21.096498 4075 leader_election.cc:304] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc96db7be7426f8dc151f7ef365234, b4691314ae524e4aaa16630b81a79cfd; no voters:
I20251028 14:52:21.096931 4496 raft_consensus.cc:2804] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:21.097076 4496 raft_consensus.cc:493] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:21.097167 4496 raft_consensus.cc:3060] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:21.099566 4496 raft_consensus.cc:515] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:21.100234 4496 leader_election.cc:290] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [CANDIDATE]: Term 1 election: Requested vote from peers b5dcbffb7bfd4af29b6e248ba2d09ac2 (127.0.104.2:43431), 52dc96db7be7426f8dc151f7ef365234 (127.0.104.3:40259)
I20251028 14:52:21.101087 4292 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3202a0eb447744999af8a7b7d3618b08" candidate_uuid: "b4691314ae524e4aaa16630b81a79cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2"
I20251028 14:52:21.101110 4428 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3202a0eb447744999af8a7b7d3618b08" candidate_uuid: "b4691314ae524e4aaa16630b81a79cfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52dc96db7be7426f8dc151f7ef365234"
I20251028 14:52:21.101323 4292 raft_consensus.cc:3060] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:21.101357 4428 raft_consensus.cc:3060] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:21.103477 4428 raft_consensus.cc:2468] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4691314ae524e4aaa16630b81a79cfd in term 1.
I20251028 14:52:21.103489 4292 raft_consensus.cc:2468] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4691314ae524e4aaa16630b81a79cfd in term 1.
I20251028 14:52:21.104046 4075 leader_election.cc:304] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52dc96db7be7426f8dc151f7ef365234, b4691314ae524e4aaa16630b81a79cfd; no voters:
I20251028 14:52:21.104387 4496 raft_consensus.cc:2804] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:21.104964 4496 raft_consensus.cc:697] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [term 1 LEADER]: Becoming Leader. State: Replica: b4691314ae524e4aaa16630b81a79cfd, State: Running, Role: LEADER
I20251028 14:52:21.105463 4496 consensus_queue.cc:237] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } }
I20251028 14:52:21.107491 4568 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: TxnStatusTablet state changed. Reason: New leader b4691314ae524e4aaa16630b81a79cfd. Latest consensus state: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:21.107859 4568 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:21.108592 4570 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:21.108867 4570 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:21.109647 3983 catalog_manager.cc:5649] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd reported cstate change: term changed from 0 to 1, leader changed from <none> to b4691314ae524e4aaa16630b81a79cfd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:21.610153 4568 consensus_queue.cc:1048] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:21.612757 4572 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: TxnStatusTablet state changed. Reason: New leader b4691314ae524e4aaa16630b81a79cfd. Latest consensus state: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:21.613014 4572 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:21.615689 4568 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:21.615988 4568 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:21.621596 4568 consensus_queue.cc:1048] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:21.622460 4572 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:21.622678 4572 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P 52dc96db7be7426f8dc151f7ef365234: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:21.624863 4504 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:21.625077 4504 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:21.625846 4499 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: TxnStatusTablet state changed. Reason: New leader b4691314ae524e4aaa16630b81a79cfd. Latest consensus state: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:21.626076 4499 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P b5dcbffb7bfd4af29b6e248ba2d09ac2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:21.676577 4574 tablet_replica.cc:442] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4691314ae524e4aaa16630b81a79cfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4691314ae524e4aaa16630b81a79cfd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39721 } } peers { permanent_uuid: "b5dcbffb7bfd4af29b6e248ba2d09ac2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43431 } } peers { permanent_uuid: "52dc96db7be7426f8dc151f7ef365234" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 40259 } } }
I20251028 14:52:21.676973 4574 tablet_replica.cc:445] T 3202a0eb447744999af8a7b7d3618b08 P b4691314ae524e4aaa16630b81a79cfd: This TxnStatusTablet replica's current role is: LEADER
W20251028 14:52:21.905781 4532 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.0.104.62:35765, user_credentials={real_user=slave}} blocked reactor thread for 65732.1us
I20251028 14:52:23.115237 416 txn_write_ops-itest.cc:831] txn write RPCs completed: 8
I20251028 14:52:23.115345 416 txn_write_ops-itest.cc:833] txn write RPC rate: 160 req/sec
I20251028 14:52:23.145658 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 4059
I20251028 14:52:23.147609 4189 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:23.280232 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 4059
I20251028 14:52:23.304491 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 4196
I20251028 14:52:23.305693 4332 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:23.449467 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 4196
I20251028 14:52:23.467661 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 4339
I20251028 14:52:23.468856 4468 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:23.571331 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 4339
I20251028 14:52:23.587039 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 3954
I20251028 14:52:23.588544 4025 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:23.703261 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 3954
2025-10-28T14:52:23Z chronyd exiting
[ OK ] TxnWriteOpsITest.WriteOpPerf (5705 ms)
[ RUN ] TxnWriteOpsITest.WriteOpForNonExistentTxn
2025-10-28T14:52:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:52:23Z Disabled control of system clock
I20251028 14:52:23.772611 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41117
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41117
--txn_manager_enabled=true with env {}
W20251028 14:52:24.132813 4660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:24.133337 4660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:24.133473 4660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:24.142918 4660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:52:24.143060 4660 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:52:24.143112 4660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:24.143153 4660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:52:24.143217 4660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:52:24.154228 4660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41117
--txn_manager_enabled=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:24.155902 4660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:24.157850 4660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:24.167626 4666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:24.167793 4665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:24.167762 4668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:24.169123 4660 server_base.cc:1047] running on GCE node
I20251028 14:52:24.170462 4660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:24.172387 4660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:24.173765 4660 hybrid_clock.cc:648] HybridClock initialized: now 1761663144173639 us; error 41 us; skew 500 ppm
I20251028 14:52:24.174242 4660 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:24.176985 4660 webserver.cc:492] Webserver started at http://127.0.104.62:38773/ using document root <none> and password file <none>
I20251028 14:52:24.177912 4660 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:24.178058 4660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:24.178438 4660 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:24.181128 4660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "cc5f54badd774d5e8dc95d0531934989"
format_stamp: "Formatted at 2025-10-28 14:52:24 on dist-test-slave-0v9m"
I20251028 14:52:24.182066 4660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "cc5f54badd774d5e8dc95d0531934989"
format_stamp: "Formatted at 2025-10-28 14:52:24 on dist-test-slave-0v9m"
I20251028 14:52:24.188501 4660 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20251028 14:52:24.192642 4674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:24.194315 4660 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:52:24.194519 4660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "cc5f54badd774d5e8dc95d0531934989"
format_stamp: "Formatted at 2025-10-28 14:52:24 on dist-test-slave-0v9m"
I20251028 14:52:24.194751 4660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:24.239008 4660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:24.239970 4660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:24.240257 4660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:24.264791 4660 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41117
I20251028 14:52:24.264822 4735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41117 every 8 connection(s)
I20251028 14:52:24.266470 4660 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:52:24.271121 4736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:24.276499 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 4660
I20251028 14:52:24.276762 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:52:24.282027 4736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989: Bootstrap starting.
I20251028 14:52:24.285990 4736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:24.287477 4736 log.cc:826] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:24.290649 4736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989: No bootstrap required, opened a new log
I20251028 14:52:24.296559 4736 raft_consensus.cc:359] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5f54badd774d5e8dc95d0531934989" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41117 } }
I20251028 14:52:24.296949 4736 raft_consensus.cc:385] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:24.297058 4736 raft_consensus.cc:740] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5f54badd774d5e8dc95d0531934989, State: Initialized, Role: FOLLOWER
I20251028 14:52:24.297744 4736 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5f54badd774d5e8dc95d0531934989" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41117 } }
I20251028 14:52:24.297956 4736 raft_consensus.cc:399] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:24.298054 4736 raft_consensus.cc:493] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:24.298179 4736 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:24.300666 4736 raft_consensus.cc:515] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5f54badd774d5e8dc95d0531934989" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41117 } }
I20251028 14:52:24.301198 4736 leader_election.cc:304] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc5f54badd774d5e8dc95d0531934989; no voters:
I20251028 14:52:24.301852 4736 leader_election.cc:290] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:24.301998 4741 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:24.302990 4741 raft_consensus.cc:697] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [term 1 LEADER]: Becoming Leader. State: Replica: cc5f54badd774d5e8dc95d0531934989, State: Running, Role: LEADER
I20251028 14:52:24.303766 4741 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5f54badd774d5e8dc95d0531934989" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41117 } }
I20251028 14:52:24.304152 4736 sys_catalog.cc:565] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:24.306794 4743 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc5f54badd774d5e8dc95d0531934989. Latest consensus state: current_term: 1 leader_uuid: "cc5f54badd774d5e8dc95d0531934989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5f54badd774d5e8dc95d0531934989" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41117 } } }
I20251028 14:52:24.307159 4743 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:24.307643 4742 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc5f54badd774d5e8dc95d0531934989" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc5f54badd774d5e8dc95d0531934989" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41117 } } }
I20251028 14:52:24.307976 4742 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:24.308202 4750 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:24.320587 4750 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:24.331697 4750 catalog_manager.cc:1357] Generated new cluster ID: 5410672a7c0744c0b56f876b7285e13c
I20251028 14:52:24.331862 4750 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:24.347081 4750 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:24.348870 4750 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:24.361366 4750 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cc5f54badd774d5e8dc95d0531934989: Generated new TSK 0
I20251028 14:52:24.362412 4750 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:24.368456 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41117
--builtin_ntp_servers=127.0.104.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:52:24.719906 4761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:24.720242 4761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:24.720356 4761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:24.729242 4761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:24.729475 4761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:24.729593 4761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:52:24.740423 4761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:41117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:24.742238 4761 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:24.744257 4761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:24.754755 4767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:24.755424 4766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:24.755745 4761 server_base.cc:1047] running on GCE node
W20251028 14:52:24.756175 4769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:24.757040 4761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:24.758337 4761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:24.759629 4761 hybrid_clock.cc:648] HybridClock initialized: now 1761663144759513 us; error 108 us; skew 500 ppm
I20251028 14:52:24.760015 4761 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:24.762953 4761 webserver.cc:492] Webserver started at http://127.0.104.1:37271/ using document root <none> and password file <none>
I20251028 14:52:24.763818 4761 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:24.763962 4761 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:24.764336 4761 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:24.767169 4761 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "205a8322a568439082b24029a6a142e0"
format_stamp: "Formatted at 2025-10-28 14:52:24 on dist-test-slave-0v9m"
I20251028 14:52:24.768075 4761 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "205a8322a568439082b24029a6a142e0"
format_stamp: "Formatted at 2025-10-28 14:52:24 on dist-test-slave-0v9m"
I20251028 14:52:24.774854 4761 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20251028 14:52:24.779320 4775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:24.780918 4761 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251028 14:52:24.781122 4761 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "205a8322a568439082b24029a6a142e0"
format_stamp: "Formatted at 2025-10-28 14:52:24 on dist-test-slave-0v9m"
I20251028 14:52:24.781371 4761 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:24.809320 4761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:24.810565 4761 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:24.810932 4761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:24.815533 4761 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:24.815696 4761 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:24.815842 4761 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:24.815948 4761 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:24.862772 4761 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:36205
I20251028 14:52:24.863013 4895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:36205 every 8 connection(s)
I20251028 14:52:24.864403 4761 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:52:24.867122 4896 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41117
I20251028 14:52:24.867450 4896 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:24.868316 4896 heartbeater.cc:507] Master 127.0.104.62:41117 requested a full tablet report, sending...
I20251028 14:52:24.870697 4691 ts_manager.cc:194] Registered new tserver with Master: 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205)
I20251028 14:52:24.872493 4691 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:54031
I20251028 14:52:24.874852 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 4761
I20251028 14:52:24.875066 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:52:24.879171 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41117
--builtin_ntp_servers=127.0.104.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:52:25.228307 4898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:25.228657 4898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:25.228781 4898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:25.237581 4898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:25.237766 4898 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:25.237852 4898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:52:25.247963 4898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:41117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:25.249711 4898 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:25.251631 4898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:25.263322 4903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:25.264549 4906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:25.265137 4904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:25.265972 4898 server_base.cc:1047] running on GCE node
I20251028 14:52:25.266640 4898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:25.267686 4898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:25.268954 4898 hybrid_clock.cc:648] HybridClock initialized: now 1761663145268867 us; error 77 us; skew 500 ppm
I20251028 14:52:25.269356 4898 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:25.272022 4898 webserver.cc:492] Webserver started at http://127.0.104.2:37575/ using document root <none> and password file <none>
I20251028 14:52:25.272851 4898 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:25.272998 4898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:25.273434 4898 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:25.276127 4898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "9573bce7a5db4426ad908ea15748b25a"
format_stamp: "Formatted at 2025-10-28 14:52:25 on dist-test-slave-0v9m"
I20251028 14:52:25.277015 4898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "9573bce7a5db4426ad908ea15748b25a"
format_stamp: "Formatted at 2025-10-28 14:52:25 on dist-test-slave-0v9m"
I20251028 14:52:25.283577 4898 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251028 14:52:25.288122 4912 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:25.289834 4898 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251028 14:52:25.290052 4898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "9573bce7a5db4426ad908ea15748b25a"
format_stamp: "Formatted at 2025-10-28 14:52:25 on dist-test-slave-0v9m"
I20251028 14:52:25.290307 4898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:25.329360 4898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:25.330346 4898 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:25.330642 4898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:25.334355 4898 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:25.334489 4898 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:25.334611 4898 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:25.334707 4898 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:25.380244 4898 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42399
I20251028 14:52:25.380302 5031 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42399 every 8 connection(s)
I20251028 14:52:25.381886 4898 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:52:25.382790 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 4898
I20251028 14:52:25.383014 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:52:25.384594 5032 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41117
I20251028 14:52:25.384927 5032 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:25.385895 5032 heartbeater.cc:507] Master 127.0.104.62:41117 requested a full tablet report, sending...
I20251028 14:52:25.386472 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41117
--builtin_ntp_servers=127.0.104.20:39697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:52:25.387494 4690 ts_manager.cc:194] Registered new tserver with Master: 9573bce7a5db4426ad908ea15748b25a (127.0.104.2:42399)
I20251028 14:52:25.388366 4690 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:54311
W20251028 14:52:25.737970 5034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:25.738306 5034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:25.738394 5034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:25.747156 5034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:25.747306 5034 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:25.747393 5034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:52:25.758919 5034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:41117
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:25.760588 5034 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:25.762560 5034 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:25.773047 5039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:25.773092 5040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:25.773486 5042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:25.774596 5034 server_base.cc:1047] running on GCE node
I20251028 14:52:25.775292 5034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:25.776402 5034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:25.777873 5034 hybrid_clock.cc:648] HybridClock initialized: now 1761663145777717 us; error 126 us; skew 500 ppm
I20251028 14:52:25.778216 5034 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:25.780948 5034 webserver.cc:492] Webserver started at http://127.0.104.3:45189/ using document root <none> and password file <none>
I20251028 14:52:25.781776 5034 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:25.781904 5034 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:25.782271 5034 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:25.784826 5034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "bda14500c70e40f58c5761c6577c8397"
format_stamp: "Formatted at 2025-10-28 14:52:25 on dist-test-slave-0v9m"
I20251028 14:52:25.785750 5034 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "bda14500c70e40f58c5761c6577c8397"
format_stamp: "Formatted at 2025-10-28 14:52:25 on dist-test-slave-0v9m"
I20251028 14:52:25.792186 5034 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251028 14:52:25.796583 5048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:25.798255 5034 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:25.798444 5034 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "bda14500c70e40f58c5761c6577c8397"
format_stamp: "Formatted at 2025-10-28 14:52:25 on dist-test-slave-0v9m"
I20251028 14:52:25.798646 5034 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:25.820389 5034 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:25.821404 5034 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:25.821681 5034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:25.824800 5034 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:25.824923 5034 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:25.825042 5034 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:25.825134 5034 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:25.875522 5034 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46245
I20251028 14:52:25.875742 5167 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46245 every 8 connection(s)
I20251028 14:52:25.876324 4896 heartbeater.cc:499] Master 127.0.104.62:41117 was elected leader, sending a full tablet report...
I20251028 14:52:25.877328 5034 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:52:25.878487 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 5034
I20251028 14:52:25.878665 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:52:25.880303 5168 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41117
I20251028 14:52:25.880579 5168 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:25.881639 5168 heartbeater.cc:507] Master 127.0.104.62:41117 requested a full tablet report, sending...
I20251028 14:52:25.883296 4690 ts_manager.cc:194] Registered new tserver with Master: bda14500c70e40f58c5761c6577c8397 (127.0.104.3:46245)
I20251028 14:52:25.884140 4690 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:50541
I20251028 14:52:25.895234 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:52:25.912438 4690 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52978:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:52:25.917445 4690 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:25.950762 5103 tablet_service.cc:1505] Processing CreateTablet for tablet 30368ac2d7f645559fc3f72e2046d108 (DEFAULT_TABLE table=txn_write_ops_test [id=618409df09514eae861d7da1b3b5d66c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:25.950719 4830 tablet_service.cc:1505] Processing CreateTablet for tablet b4e9fcededd74b45be2114c85af56570 (DEFAULT_TABLE table=txn_write_ops_test [id=618409df09514eae861d7da1b3b5d66c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:25.950704 4966 tablet_service.cc:1505] Processing CreateTablet for tablet 30368ac2d7f645559fc3f72e2046d108 (DEFAULT_TABLE table=txn_write_ops_test [id=618409df09514eae861d7da1b3b5d66c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:25.952189 5103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30368ac2d7f645559fc3f72e2046d108. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:25.952206 4966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30368ac2d7f645559fc3f72e2046d108. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:25.952190 4830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e9fcededd74b45be2114c85af56570. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:25.953883 5102 tablet_service.cc:1505] Processing CreateTablet for tablet b4e9fcededd74b45be2114c85af56570 (DEFAULT_TABLE table=txn_write_ops_test [id=618409df09514eae861d7da1b3b5d66c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:25.954497 5102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e9fcededd74b45be2114c85af56570. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:25.956622 4965 tablet_service.cc:1505] Processing CreateTablet for tablet b4e9fcededd74b45be2114c85af56570 (DEFAULT_TABLE table=txn_write_ops_test [id=618409df09514eae861d7da1b3b5d66c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:25.957219 4965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4e9fcededd74b45be2114c85af56570. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:25.957850 4831 tablet_service.cc:1505] Processing CreateTablet for tablet 30368ac2d7f645559fc3f72e2046d108 (DEFAULT_TABLE table=txn_write_ops_test [id=618409df09514eae861d7da1b3b5d66c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:25.958401 4831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30368ac2d7f645559fc3f72e2046d108. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:25.974586 5186 tablet_bootstrap.cc:492] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a: Bootstrap starting.
I20251028 14:52:25.980792 5186 tablet_bootstrap.cc:654] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:25.982671 5186 log.cc:826] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:25.982998 5185 tablet_bootstrap.cc:492] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397: Bootstrap starting.
I20251028 14:52:25.984282 5187 tablet_bootstrap.cc:492] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0: Bootstrap starting.
I20251028 14:52:25.986181 5186 tablet_bootstrap.cc:492] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a: No bootstrap required, opened a new log
I20251028 14:52:25.986515 5186 ts_tablet_manager.cc:1403] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20251028 14:52:25.986637 5185 tablet_bootstrap.cc:654] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:25.987886 5185 log.cc:826] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:25.988065 5187 tablet_bootstrap.cc:654] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:25.989465 5187 log.cc:826] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:25.991277 5185 tablet_bootstrap.cc:492] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397: No bootstrap required, opened a new log
I20251028 14:52:25.991693 5185 ts_tablet_manager.cc:1403] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20251028 14:52:25.994240 5187 tablet_bootstrap.cc:492] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0: No bootstrap required, opened a new log
I20251028 14:52:25.993959 5186 raft_consensus.cc:359] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:25.994407 5186 raft_consensus.cc:385] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:25.994553 5186 raft_consensus.cc:740] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9573bce7a5db4426ad908ea15748b25a, State: Initialized, Role: FOLLOWER
I20251028 14:52:25.994652 5187 ts_tablet_manager.cc:1403] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20251028 14:52:25.995467 5186 consensus_queue.cc:260] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:25.996898 5032 heartbeater.cc:499] Master 127.0.104.62:41117 was elected leader, sending a full tablet report...
I20251028 14:52:25.997682 5185 raft_consensus.cc:359] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:25.998261 5185 raft_consensus.cc:385] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:25.998288 5186 ts_tablet_manager.cc:1434] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20251028 14:52:25.998445 5185 raft_consensus.cc:740] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda14500c70e40f58c5761c6577c8397, State: Initialized, Role: FOLLOWER
I20251028 14:52:25.998819 5186 tablet_bootstrap.cc:492] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a: Bootstrap starting.
I20251028 14:52:25.999719 5185 consensus_queue.cc:260] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.001786 5186 tablet_bootstrap.cc:654] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:26.002568 5187 raft_consensus.cc:359] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.003093 5187 raft_consensus.cc:385] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:26.003259 5187 raft_consensus.cc:740] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205a8322a568439082b24029a6a142e0, State: Initialized, Role: FOLLOWER
I20251028 14:52:26.004292 5187 consensus_queue.cc:260] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.007969 5187 ts_tablet_manager.cc:1434] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20251028 14:52:26.009282 5187 tablet_bootstrap.cc:492] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0: Bootstrap starting.
I20251028 14:52:26.011425 5187 tablet_bootstrap.cc:654] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:26.012503 5168 heartbeater.cc:499] Master 127.0.104.62:41117 was elected leader, sending a full tablet report...
I20251028 14:52:26.013985 5185 ts_tablet_manager.cc:1434] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397: Time spent starting tablet: real 0.022s user 0.013s sys 0.008s
I20251028 14:52:26.014587 5185 tablet_bootstrap.cc:492] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397: Bootstrap starting.
I20251028 14:52:26.016605 5185 tablet_bootstrap.cc:654] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:26.016690 5186 tablet_bootstrap.cc:492] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a: No bootstrap required, opened a new log
I20251028 14:52:26.016953 5186 ts_tablet_manager.cc:1403] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.009s
I20251028 14:52:26.017002 5187 tablet_bootstrap.cc:492] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0: No bootstrap required, opened a new log
I20251028 14:52:26.017223 5187 ts_tablet_manager.cc:1403] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251028 14:52:26.017944 5186 raft_consensus.cc:359] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.018221 5186 raft_consensus.cc:385] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:26.018201 5187 raft_consensus.cc:359] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.018357 5186 raft_consensus.cc:740] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9573bce7a5db4426ad908ea15748b25a, State: Initialized, Role: FOLLOWER
I20251028 14:52:26.018492 5187 raft_consensus.cc:385] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:26.018610 5187 raft_consensus.cc:740] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205a8322a568439082b24029a6a142e0, State: Initialized, Role: FOLLOWER
I20251028 14:52:26.018792 5186 consensus_queue.cc:260] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.018937 5187 consensus_queue.cc:260] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.019549 5187 ts_tablet_manager.cc:1434] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:52:26.020298 5186 ts_tablet_manager.cc:1434] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251028 14:52:26.022256 5185 tablet_bootstrap.cc:492] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397: No bootstrap required, opened a new log
I20251028 14:52:26.022579 5185 ts_tablet_manager.cc:1403] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20251028 14:52:26.023900 5185 raft_consensus.cc:359] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.024099 5185 raft_consensus.cc:385] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:26.024207 5185 raft_consensus.cc:740] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda14500c70e40f58c5761c6577c8397, State: Initialized, Role: FOLLOWER
I20251028 14:52:26.024444 5185 consensus_queue.cc:260] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.025144 5185 ts_tablet_manager.cc:1434] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20251028 14:52:26.117338 4897 tablet.cc:2378] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:26.129381 5169 tablet.cc:2378] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:26.134204 5033 tablet.cc:2378] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:52:26.167078 5191 raft_consensus.cc:493] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:26.167354 5191 raft_consensus.cc:515] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.169214 5191 leader_election.cc:290] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205), bda14500c70e40f58c5761c6577c8397 (127.0.104.3:46245)
I20251028 14:52:26.176576 5123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4e9fcededd74b45be2114c85af56570" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda14500c70e40f58c5761c6577c8397" is_pre_election: true
I20251028 14:52:26.176652 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4e9fcededd74b45be2114c85af56570" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0" is_pre_election: true
I20251028 14:52:26.177100 4851 raft_consensus.cc:2468] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 0.
I20251028 14:52:26.177094 5123 raft_consensus.cc:2468] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 0.
I20251028 14:52:26.177824 4914 leader_election.cc:304] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9573bce7a5db4426ad908ea15748b25a, bda14500c70e40f58c5761c6577c8397; no voters:
I20251028 14:52:26.178233 5191 raft_consensus.cc:2804] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:26.178377 5191 raft_consensus.cc:493] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:26.178475 5191 raft_consensus.cc:3060] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.180662 5191 raft_consensus.cc:515] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.181326 5191 leader_election.cc:290] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 election: Requested vote from peers 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205), bda14500c70e40f58c5761c6577c8397 (127.0.104.3:46245)
I20251028 14:52:26.181968 5123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4e9fcededd74b45be2114c85af56570" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda14500c70e40f58c5761c6577c8397"
I20251028 14:52:26.181968 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b4e9fcededd74b45be2114c85af56570" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0"
I20251028 14:52:26.182188 4851 raft_consensus.cc:3060] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.182188 5123 raft_consensus.cc:3060] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.185566 5123 raft_consensus.cc:2468] T b4e9fcededd74b45be2114c85af56570 P bda14500c70e40f58c5761c6577c8397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 1.
I20251028 14:52:26.185566 4851 raft_consensus.cc:2468] T b4e9fcededd74b45be2114c85af56570 P 205a8322a568439082b24029a6a142e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 1.
I20251028 14:52:26.186086 4914 leader_election.cc:304] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 205a8322a568439082b24029a6a142e0, 9573bce7a5db4426ad908ea15748b25a, bda14500c70e40f58c5761c6577c8397; no voters:
I20251028 14:52:26.186447 5191 raft_consensus.cc:2804] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:26.186903 5191 raft_consensus.cc:697] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [term 1 LEADER]: Becoming Leader. State: Replica: 9573bce7a5db4426ad908ea15748b25a, State: Running, Role: LEADER
I20251028 14:52:26.187423 5191 consensus_queue.cc:237] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.192955 4691 catalog_manager.cc:5649] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9573bce7a5db4426ad908ea15748b25a (127.0.104.2). New cstate: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:26.299299 5192 raft_consensus.cc:493] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:26.299541 5192 raft_consensus.cc:515] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.301367 5192 leader_election.cc:290] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205), 9573bce7a5db4426ad908ea15748b25a (127.0.104.2:42399)
I20251028 14:52:26.303252 5191 raft_consensus.cc:493] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:26.303473 5191 raft_consensus.cc:515] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.304296 5191 leader_election.cc:290] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bda14500c70e40f58c5761c6577c8397 (127.0.104.3:46245), 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205)
I20251028 14:52:26.304708 5123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda14500c70e40f58c5761c6577c8397" is_pre_election: true
I20251028 14:52:26.304978 5123 raft_consensus.cc:2468] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 0.
I20251028 14:52:26.305106 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0" is_pre_election: true
I20251028 14:52:26.305393 4851 raft_consensus.cc:2468] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 0.
I20251028 14:52:26.305608 4914 leader_election.cc:304] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9573bce7a5db4426ad908ea15748b25a, bda14500c70e40f58c5761c6577c8397; no voters:
I20251028 14:52:26.305992 5191 raft_consensus.cc:2804] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:26.306134 5191 raft_consensus.cc:493] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:26.306233 5191 raft_consensus.cc:3060] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.308741 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "bda14500c70e40f58c5761c6577c8397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0" is_pre_election: true
I20251028 14:52:26.309056 4851 raft_consensus.cc:2468] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bda14500c70e40f58c5761c6577c8397 in term 0.
I20251028 14:52:26.309036 5191 raft_consensus.cc:515] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.309767 5191 leader_election.cc:290] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 election: Requested vote from peers bda14500c70e40f58c5761c6577c8397 (127.0.104.3:46245), 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205)
I20251028 14:52:26.309708 5049 leader_election.cc:304] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205a8322a568439082b24029a6a142e0, bda14500c70e40f58c5761c6577c8397; no voters:
I20251028 14:52:26.309633 4987 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "bda14500c70e40f58c5761c6577c8397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9573bce7a5db4426ad908ea15748b25a" is_pre_election: true
I20251028 14:52:26.310190 4987 raft_consensus.cc:2393] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bda14500c70e40f58c5761c6577c8397 in current term 1: Already voted for candidate 9573bce7a5db4426ad908ea15748b25a in this term.
I20251028 14:52:26.310179 5192 raft_consensus.cc:2804] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:26.310307 5192 raft_consensus.cc:493] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:26.310413 5192 raft_consensus.cc:3060] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.310473 5123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda14500c70e40f58c5761c6577c8397"
I20251028 14:52:26.310600 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0"
I20251028 14:52:26.310819 4851 raft_consensus.cc:3060] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.313699 4851 raft_consensus.cc:2468] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 1.
I20251028 14:52:26.313730 5192 raft_consensus.cc:515] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.314200 5123 raft_consensus.cc:2393] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9573bce7a5db4426ad908ea15748b25a in current term 1: Already voted for candidate bda14500c70e40f58c5761c6577c8397 in this term.
I20251028 14:52:26.314271 4913 leader_election.cc:304] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205a8322a568439082b24029a6a142e0, 9573bce7a5db4426ad908ea15748b25a; no voters:
I20251028 14:52:26.314536 5192 leader_election.cc:290] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [CANDIDATE]: Term 1 election: Requested vote from peers 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205), 9573bce7a5db4426ad908ea15748b25a (127.0.104.2:42399)
I20251028 14:52:26.315092 5191 raft_consensus.cc:2804] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:26.315052 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "bda14500c70e40f58c5761c6577c8397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0"
I20251028 14:52:26.315171 4987 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30368ac2d7f645559fc3f72e2046d108" candidate_uuid: "bda14500c70e40f58c5761c6577c8397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9573bce7a5db4426ad908ea15748b25a"
I20251028 14:52:26.315315 5191 raft_consensus.cc:697] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [term 1 LEADER]: Becoming Leader. State: Replica: 9573bce7a5db4426ad908ea15748b25a, State: Running, Role: LEADER
I20251028 14:52:26.315379 4851 raft_consensus.cc:2393] T 30368ac2d7f645559fc3f72e2046d108 P 205a8322a568439082b24029a6a142e0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bda14500c70e40f58c5761c6577c8397 in current term 1: Already voted for candidate 9573bce7a5db4426ad908ea15748b25a in this term.
I20251028 14:52:26.315626 5191 consensus_queue.cc:237] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } }
I20251028 14:52:26.318043 5050 leader_election.cc:304] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bda14500c70e40f58c5761c6577c8397; no voters: 205a8322a568439082b24029a6a142e0, 9573bce7a5db4426ad908ea15748b25a
I20251028 14:52:26.318534 5192 raft_consensus.cc:2749] T 30368ac2d7f645559fc3f72e2046d108 P bda14500c70e40f58c5761c6577c8397 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251028 14:52:26.319550 4691 catalog_manager.cc:5649] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9573bce7a5db4426ad908ea15748b25a (127.0.104.2). New cstate: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:26.469027 4691 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52980:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:26.470220 4691 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:26.478607 4965 tablet_service.cc:1505] Processing CreateTablet for tablet b88229bfff264ce28d0b95c88e0c6f58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f63a8fd26544302b82a4af0498031d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:26.478482 4830 tablet_service.cc:1505] Processing CreateTablet for tablet b88229bfff264ce28d0b95c88e0c6f58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f63a8fd26544302b82a4af0498031d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:26.478912 5102 tablet_service.cc:1505] Processing CreateTablet for tablet b88229bfff264ce28d0b95c88e0c6f58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f63a8fd26544302b82a4af0498031d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:26.479533 4965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88229bfff264ce28d0b95c88e0c6f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:26.479534 4830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88229bfff264ce28d0b95c88e0c6f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:26.479653 5102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b88229bfff264ce28d0b95c88e0c6f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:26.485816 5186 tablet_bootstrap.cc:492] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: Bootstrap starting.
I20251028 14:52:26.487206 5187 tablet_bootstrap.cc:492] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: Bootstrap starting.
I20251028 14:52:26.487689 5186 tablet_bootstrap.cc:654] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:26.487763 5185 tablet_bootstrap.cc:492] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: Bootstrap starting.
I20251028 14:52:26.489081 5187 tablet_bootstrap.cc:654] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:26.489575 5185 tablet_bootstrap.cc:654] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:26.491572 5185 tablet_bootstrap.cc:492] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: No bootstrap required, opened a new log
I20251028 14:52:26.491669 5187 tablet_bootstrap.cc:492] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: No bootstrap required, opened a new log
I20251028 14:52:26.491674 5186 tablet_bootstrap.cc:492] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: No bootstrap required, opened a new log
I20251028 14:52:26.491761 5185 ts_tablet_manager.cc:1403] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:26.491859 5187 ts_tablet_manager.cc:1403] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:52:26.491861 5186 ts_tablet_manager.cc:1403] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20251028 14:52:26.492586 5185 raft_consensus.cc:359] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.492791 5185 raft_consensus.cc:385] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:26.492871 5185 raft_consensus.cc:740] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bda14500c70e40f58c5761c6577c8397, State: Initialized, Role: FOLLOWER
I20251028 14:52:26.493184 5187 raft_consensus.cc:359] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.493440 5187 raft_consensus.cc:385] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:26.493507 5187 raft_consensus.cc:740] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 205a8322a568439082b24029a6a142e0, State: Initialized, Role: FOLLOWER
I20251028 14:52:26.493403 5186 raft_consensus.cc:359] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.493392 5185 consensus_queue.cc:260] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.493621 5186 raft_consensus.cc:385] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:26.493757 5186 raft_consensus.cc:740] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9573bce7a5db4426ad908ea15748b25a, State: Initialized, Role: FOLLOWER
I20251028 14:52:26.493839 5187 consensus_queue.cc:260] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.494014 5186 consensus_queue.cc:260] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.494114 5185 ts_tablet_manager.cc:1434] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:26.494408 5187 ts_tablet_manager.cc:1434] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:52:26.494448 5191 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.494655 5191 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:26.494488 5194 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.494834 5194 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:26.495170 5186 ts_tablet_manager.cc:1434] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251028 14:52:26.496122 5192 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.496441 5192 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:26.668008 5191 raft_consensus.cc:493] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:26.668222 5191 raft_consensus.cc:515] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.668886 5191 leader_election.cc:290] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205), bda14500c70e40f58c5761c6577c8397 (127.0.104.3:46245)
I20251028 14:52:26.669528 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88229bfff264ce28d0b95c88e0c6f58" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0" is_pre_election: true
I20251028 14:52:26.669507 5123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88229bfff264ce28d0b95c88e0c6f58" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda14500c70e40f58c5761c6577c8397" is_pre_election: true
I20251028 14:52:26.669797 5123 raft_consensus.cc:2468] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 0.
I20251028 14:52:26.669775 4851 raft_consensus.cc:2468] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 0.
I20251028 14:52:26.670281 4913 leader_election.cc:304] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205a8322a568439082b24029a6a142e0, 9573bce7a5db4426ad908ea15748b25a; no voters:
I20251028 14:52:26.670563 5191 raft_consensus.cc:2804] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:26.670679 5191 raft_consensus.cc:493] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:26.670750 5191 raft_consensus.cc:3060] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.672739 5191 raft_consensus.cc:515] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.673669 5191 leader_election.cc:290] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 election: Requested vote from peers 205a8322a568439082b24029a6a142e0 (127.0.104.1:36205), bda14500c70e40f58c5761c6577c8397 (127.0.104.3:46245)
I20251028 14:52:26.674157 4851 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88229bfff264ce28d0b95c88e0c6f58" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "205a8322a568439082b24029a6a142e0"
I20251028 14:52:26.674353 4851 raft_consensus.cc:3060] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.674315 5123 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b88229bfff264ce28d0b95c88e0c6f58" candidate_uuid: "9573bce7a5db4426ad908ea15748b25a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bda14500c70e40f58c5761c6577c8397"
I20251028 14:52:26.674635 5123 raft_consensus.cc:3060] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:26.676589 4851 raft_consensus.cc:2468] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 1.
I20251028 14:52:26.676941 5123 raft_consensus.cc:2468] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9573bce7a5db4426ad908ea15748b25a in term 1.
I20251028 14:52:26.677165 4913 leader_election.cc:304] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 205a8322a568439082b24029a6a142e0, 9573bce7a5db4426ad908ea15748b25a; no voters:
I20251028 14:52:26.677565 5191 raft_consensus.cc:2804] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:26.677733 5191 raft_consensus.cc:697] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [term 1 LEADER]: Becoming Leader. State: Replica: 9573bce7a5db4426ad908ea15748b25a, State: Running, Role: LEADER
I20251028 14:52:26.678030 5191 consensus_queue.cc:237] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } }
I20251028 14:52:26.679523 5199 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: TxnStatusTablet state changed. Reason: New leader 9573bce7a5db4426ad908ea15748b25a. Latest consensus state: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.679842 5199 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:26.680447 5215 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:26.680734 5215 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:26.680945 4690 catalog_manager.cc:5649] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9573bce7a5db4426ad908ea15748b25a (127.0.104.2). New cstate: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:26.727540 5199 consensus_queue.cc:1048] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:26.750732 5191 consensus_queue.cc:1048] T 30368ac2d7f645559fc3f72e2046d108 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:26.758901 5199 consensus_queue.cc:1048] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:26.765666 5122 raft_consensus.cc:1275] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397 [term 1 FOLLOWER]: Refusing update from remote peer 9573bce7a5db4426ad908ea15748b25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:26.765846 4851 raft_consensus.cc:1275] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0 [term 1 FOLLOWER]: Refusing update from remote peer 9573bce7a5db4426ad908ea15748b25a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:26.766423 5191 consensus_queue.cc:1048] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:26.766966 5199 consensus_queue.cc:1048] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:26.769433 5199 consensus_queue.cc:1048] T b4e9fcededd74b45be2114c85af56570 P 9573bce7a5db4426ad908ea15748b25a [LEADER]: Connected to new peer: Peer: permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:26.781924 5194 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: TxnStatusTablet state changed. Reason: New leader 9573bce7a5db4426ad908ea15748b25a. Latest consensus state: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.782214 5194 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:26.785670 5192 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: TxnStatusTablet state changed. Reason: New leader 9573bce7a5db4426ad908ea15748b25a. Latest consensus state: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.785988 5192 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:26.788266 5218 mvcc.cc:204] Tried to move back new op lower bound from 7215772249143861248 to 7215772248795308032. Current Snapshot: MvccSnapshot[applied={T|T < 7215772249143861248}]
I20251028 14:52:26.807448 5224 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.808655 5224 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:26.809646 5199 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.809865 5199 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P 9573bce7a5db4426ad908ea15748b25a: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:26.813779 5194 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.814018 5194 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P 205a8322a568439082b24029a6a142e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:26.818419 5192 tablet_replica.cc:442] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9573bce7a5db4426ad908ea15748b25a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "205a8322a568439082b24029a6a142e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36205 } } peers { permanent_uuid: "9573bce7a5db4426ad908ea15748b25a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42399 } } peers { permanent_uuid: "bda14500c70e40f58c5761c6577c8397" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46245 } } }
I20251028 14:52:26.818674 5192 tablet_replica.cc:445] T b88229bfff264ce28d0b95c88e0c6f58 P bda14500c70e40f58c5761c6577c8397: This TxnStatusTablet replica's current role is: FOLLOWER
W20251028 14:52:26.855204 5237 tablet_replica.cc:1306] Invalid argument: Failed to write to server: 9573bce7a5db4426ad908ea15748b25a (127.0.104.2:42399): transaction ID 100 not found, current highest txn ID: 0: cancelling pending write operations
W20251028 14:52:26.855500 5237 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39650: Invalid argument: Failed to write to server: 9573bce7a5db4426ad908ea15748b25a (127.0.104.2:42399): transaction ID 100 not found, current highest txn ID: 0
W20251028 14:52:26.856676 5171 batcher.cc:441] Invalid argument: Failed to write batch of 1 ops to tablet b4e9fcededd74b45be2114c85af56570 after 1 attempt(s): Failed to write to server: 9573bce7a5db4426ad908ea15748b25a (127.0.104.2:42399): Failed to write to server: 9573bce7a5db4426ad908ea15748b25a (127.0.104.2:42399): transaction ID 100 not found, current highest txn ID: 0
I20251028 14:52:26.857290 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 4761
I20251028 14:52:26.858372 4891 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:26.953751 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 4761
I20251028 14:52:26.970808 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 4898
I20251028 14:52:26.972048 5027 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:27.093124 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 4898
I20251028 14:52:27.109946 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 5034
I20251028 14:52:27.111018 5163 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:27.206660 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 5034
I20251028 14:52:27.221452 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 4660
I20251028 14:52:27.222563 4731 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:27.337134 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 4660
2025-10-28T14:52:27Z chronyd exiting
[ OK ] TxnWriteOpsITest.WriteOpForNonExistentTxn (3634 ms)
[ RUN ] TxnWriteOpsITest.TxnWriteAfterCommit
2025-10-28T14:52:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-28T14:52:27Z Disabled control of system clock
I20251028 14:52:27.408509 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38665
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38665
--txn_manager_enabled=true with env {}
W20251028 14:52:27.754989 5247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:27.755344 5247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:27.755421 5247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:27.764122 5247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251028 14:52:27.764249 5247 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251028 14:52:27.764299 5247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:27.764338 5247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251028 14:52:27.764403 5247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251028 14:52:27.774895 5247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38665
--txn_manager_enabled=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:27.777029 5247 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:27.779138 5247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:27.789232 5252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:27.789270 5253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:27.789700 5255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:27.790428 5247 server_base.cc:1047] running on GCE node
I20251028 14:52:27.791466 5247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:27.793332 5247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:27.794663 5247 hybrid_clock.cc:648] HybridClock initialized: now 1761663147794595 us; error 44 us; skew 500 ppm
I20251028 14:52:27.795065 5247 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:27.798301 5247 webserver.cc:492] Webserver started at http://127.0.104.62:46417/ using document root <none> and password file <none>
I20251028 14:52:27.799374 5247 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:27.799567 5247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:27.800160 5247 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:27.803002 5247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data/instance:
uuid: "0bb925a1cdc74cc2b0982632918b48a1"
format_stamp: "Formatted at 2025-10-28 14:52:27 on dist-test-slave-0v9m"
I20251028 14:52:27.803951 5247 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/wal/instance:
uuid: "0bb925a1cdc74cc2b0982632918b48a1"
format_stamp: "Formatted at 2025-10-28 14:52:27 on dist-test-slave-0v9m"
I20251028 14:52:27.810758 5247 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20251028 14:52:27.815069 5261 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:27.816706 5247 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20251028 14:52:27.816928 5247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/wal
uuid: "0bb925a1cdc74cc2b0982632918b48a1"
format_stamp: "Formatted at 2025-10-28 14:52:27 on dist-test-slave-0v9m"
I20251028 14:52:27.817149 5247 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:27.848685 5247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:27.849745 5247 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:27.850060 5247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:27.874619 5247 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38665
I20251028 14:52:27.874639 5322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38665 every 8 connection(s)
I20251028 14:52:27.876477 5247 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/data/info.pb
I20251028 14:52:27.881219 5323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:27.881877 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 5247
I20251028 14:52:27.882102 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/master-0/wal/instance
I20251028 14:52:27.894712 5323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1: Bootstrap starting.
I20251028 14:52:27.898367 5323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:27.899423 5323 log.cc:826] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:27.902364 5323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1: No bootstrap required, opened a new log
I20251028 14:52:27.908062 5323 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb925a1cdc74cc2b0982632918b48a1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38665 } }
I20251028 14:52:27.908461 5323 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:27.908542 5323 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bb925a1cdc74cc2b0982632918b48a1, State: Initialized, Role: FOLLOWER
I20251028 14:52:27.909232 5323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb925a1cdc74cc2b0982632918b48a1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38665 } }
I20251028 14:52:27.909528 5323 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:27.909711 5323 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:27.909900 5323 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:27.912096 5323 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb925a1cdc74cc2b0982632918b48a1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38665 } }
I20251028 14:52:27.912595 5323 leader_election.cc:304] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bb925a1cdc74cc2b0982632918b48a1; no voters:
I20251028 14:52:27.913169 5323 leader_election.cc:290] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:27.913365 5328 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:27.913987 5328 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [term 1 LEADER]: Becoming Leader. State: Replica: 0bb925a1cdc74cc2b0982632918b48a1, State: Running, Role: LEADER
I20251028 14:52:27.914543 5328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb925a1cdc74cc2b0982632918b48a1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38665 } }
I20251028 14:52:27.915242 5323 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:27.918744 5329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bb925a1cdc74cc2b0982632918b48a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb925a1cdc74cc2b0982632918b48a1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38665 } } }
I20251028 14:52:27.917822 5330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bb925a1cdc74cc2b0982632918b48a1. Latest consensus state: current_term: 1 leader_uuid: "0bb925a1cdc74cc2b0982632918b48a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bb925a1cdc74cc2b0982632918b48a1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38665 } } }
I20251028 14:52:27.919451 5330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:27.920008 5329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:27.920485 5337 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:27.926503 5337 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:27.936582 5337 catalog_manager.cc:1357] Generated new cluster ID: 85229d1c43a84620b4a1023ea56cc07b
I20251028 14:52:27.936728 5337 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:27.963346 5337 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:27.965197 5337 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:27.983243 5337 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0bb925a1cdc74cc2b0982632918b48a1: Generated new TSK 0
I20251028 14:52:27.984433 5337 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:27.993000 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38665
--builtin_ntp_servers=127.0.104.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:52:28.352068 5347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:28.352398 5347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:28.352519 5347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:28.361398 5347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:28.361557 5347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:28.361681 5347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:52:28.372526 5347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:38665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:28.375264 5347 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:28.377517 5347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:28.387817 5352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:28.388130 5353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:28.388162 5355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:28.389106 5347 server_base.cc:1047] running on GCE node
I20251028 14:52:28.389776 5347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:28.390820 5347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:28.392078 5347 hybrid_clock.cc:648] HybridClock initialized: now 1761663148391974 us; error 65 us; skew 500 ppm
I20251028 14:52:28.392395 5347 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:28.395153 5347 webserver.cc:492] Webserver started at http://127.0.104.1:44467/ using document root <none> and password file <none>
I20251028 14:52:28.396083 5347 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:28.396206 5347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:28.396554 5347 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:28.399391 5347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/instance:
uuid: "852e2c5586bc46ca81ae46d74d4ed7fa"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:28.400246 5347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal/instance:
uuid: "852e2c5586bc46ca81ae46d74d4ed7fa"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:28.406431 5347 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20251028 14:52:28.410840 5361 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:28.412982 5347 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20251028 14:52:28.413213 5347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "852e2c5586bc46ca81ae46d74d4ed7fa"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:28.413731 5347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:28.431068 5347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:28.432508 5347 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:28.432794 5347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:28.436229 5347 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:28.436360 5347 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:28.436486 5347 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:28.436578 5347 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:28.487471 5347 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:41875
I20251028 14:52:28.487536 5481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:41875 every 8 connection(s)
I20251028 14:52:28.489027 5347 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
I20251028 14:52:28.492273 5482 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38665
I20251028 14:52:28.492635 5482 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:28.493613 5482 heartbeater.cc:507] Master 127.0.104.62:38665 requested a full tablet report, sending...
I20251028 14:52:28.496331 5278 ts_manager.cc:194] Registered new tserver with Master: 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875)
I20251028 14:52:28.496466 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 5347
I20251028 14:52:28.496696 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal/instance
I20251028 14:52:28.498548 5278 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:33049
I20251028 14:52:28.501864 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38665
--builtin_ntp_servers=127.0.104.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:52:28.855278 5484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:28.855636 5484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:28.855763 5484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:28.864563 5484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:28.864741 5484 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:28.864864 5484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:52:28.875439 5484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:38665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:28.877221 5484 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:28.879236 5484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:28.890157 5489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:28.890902 5490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:28.892676 5492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:28.893090 5484 server_base.cc:1047] running on GCE node
I20251028 14:52:28.893977 5484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:28.895220 5484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:28.898715 5484 hybrid_clock.cc:648] HybridClock initialized: now 1761663148898518 us; error 153 us; skew 500 ppm
I20251028 14:52:28.899132 5484 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:28.901943 5484 webserver.cc:492] Webserver started at http://127.0.104.2:44665/ using document root <none> and password file <none>
I20251028 14:52:28.902794 5484 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:28.902940 5484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:28.903307 5484 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:28.906001 5484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/instance:
uuid: "b76b7e0f1dd442798012ff773075f578"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:28.906916 5484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal/instance:
uuid: "b76b7e0f1dd442798012ff773075f578"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:28.913093 5484 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20251028 14:52:28.917511 5498 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:28.919384 5484 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20251028 14:52:28.919587 5484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "b76b7e0f1dd442798012ff773075f578"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:28.919824 5484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:28.935048 5484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:28.935949 5484 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:28.936223 5484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:28.939416 5484 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:28.939540 5484 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:28.939647 5484 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:28.939718 5484 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:28.991981 5484 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38183
I20251028 14:52:28.992200 5617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38183 every 8 connection(s)
I20251028 14:52:28.993533 5484 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:52:28.995007 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 5484
I20251028 14:52:28.995297 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal/instance
I20251028 14:52:28.996553 5618 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38665
I20251028 14:52:28.996878 5618 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:28.997871 5618 heartbeater.cc:507] Master 127.0.104.62:38665 requested a full tablet report, sending...
I20251028 14:52:28.998430 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38665
--builtin_ntp_servers=127.0.104.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20251028 14:52:28.999433 5277 ts_manager.cc:194] Registered new tserver with Master: b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:29.000603 5277 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:41035
W20251028 14:52:29.370565 5620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:29.370913 5620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:29.370998 5620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:29.379251 5620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:29.379406 5620 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:29.379487 5620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:52:29.389966 5620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:38665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:29.391710 5620 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:29.393649 5620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:29.404362 5626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:29.404393 5625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:29.404824 5628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:29.406036 5620 server_base.cc:1047] running on GCE node
I20251028 14:52:29.406795 5620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:29.407812 5620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:29.409063 5620 hybrid_clock.cc:648] HybridClock initialized: now 1761663149408971 us; error 64 us; skew 500 ppm
I20251028 14:52:29.409458 5620 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:29.412717 5620 webserver.cc:492] Webserver started at http://127.0.104.3:46041/ using document root <none> and password file <none>
I20251028 14:52:29.413841 5620 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:29.414157 5620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:29.414672 5620 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:29.417511 5620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/instance:
uuid: "a050795decf2492bbbefa78269460b5d"
format_stamp: "Formatted at 2025-10-28 14:52:29 on dist-test-slave-0v9m"
I20251028 14:52:29.418413 5620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal/instance:
uuid: "a050795decf2492bbbefa78269460b5d"
format_stamp: "Formatted at 2025-10-28 14:52:29 on dist-test-slave-0v9m"
I20251028 14:52:29.425503 5620 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20251028 14:52:29.429549 5634 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:29.431241 5620 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:52:29.431432 5620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "a050795decf2492bbbefa78269460b5d"
format_stamp: "Formatted at 2025-10-28 14:52:29 on dist-test-slave-0v9m"
I20251028 14:52:29.431633 5620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:29.457928 5620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:29.458807 5620 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:29.459089 5620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:29.462189 5620 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:29.462316 5620 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:29.462474 5620 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:29.462550 5620 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:29.502802 5482 heartbeater.cc:499] Master 127.0.104.62:38665 was elected leader, sending a full tablet report...
I20251028 14:52:29.505100 5620 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:33277
I20251028 14:52:29.505168 5753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:33277 every 8 connection(s)
I20251028 14:52:29.506479 5620 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:52:29.508668 5754 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38665
I20251028 14:52:29.508968 5754 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:29.509790 5754 heartbeater.cc:507] Master 127.0.104.62:38665 requested a full tablet report, sending...
I20251028 14:52:29.511304 5277 ts_manager.cc:194] Registered new tserver with Master: a050795decf2492bbbefa78269460b5d (127.0.104.3:33277)
I20251028 14:52:29.511618 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 5620
I20251028 14:52:29.511780 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal/instance
I20251028 14:52:29.512274 5277 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:57527
I20251028 14:52:29.515312 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251028 14:52:29.532625 5277 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49618:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:52:29.537740 5277 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:29.583313 5416 tablet_service.cc:1505] Processing CreateTablet for tablet aa2fc80f5ab94e9699132e99a180de02 (DEFAULT_TABLE table=txn_write_ops_test [id=68d83acac4ed423080b5a8dc22ef130e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:29.583408 5684 tablet_service.cc:1505] Processing CreateTablet for tablet aa2fc80f5ab94e9699132e99a180de02 (DEFAULT_TABLE table=txn_write_ops_test [id=68d83acac4ed423080b5a8dc22ef130e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:29.583408 5553 tablet_service.cc:1505] Processing CreateTablet for tablet aa2fc80f5ab94e9699132e99a180de02 (DEFAULT_TABLE table=txn_write_ops_test [id=68d83acac4ed423080b5a8dc22ef130e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:29.584944 5416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2fc80f5ab94e9699132e99a180de02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.584966 5684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2fc80f5ab94e9699132e99a180de02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.584975 5553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa2fc80f5ab94e9699132e99a180de02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.584941 5552 tablet_service.cc:1505] Processing CreateTablet for tablet 0b36c4279f6a484faa06c9f6c4cc06c2 (DEFAULT_TABLE table=txn_write_ops_test [id=68d83acac4ed423080b5a8dc22ef130e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:29.585618 5415 tablet_service.cc:1505] Processing CreateTablet for tablet 0b36c4279f6a484faa06c9f6c4cc06c2 (DEFAULT_TABLE table=txn_write_ops_test [id=68d83acac4ed423080b5a8dc22ef130e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:29.585872 5552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b36c4279f6a484faa06c9f6c4cc06c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.586112 5415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b36c4279f6a484faa06c9f6c4cc06c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.589859 5683 tablet_service.cc:1505] Processing CreateTablet for tablet 0b36c4279f6a484faa06c9f6c4cc06c2 (DEFAULT_TABLE table=txn_write_ops_test [id=68d83acac4ed423080b5a8dc22ef130e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:29.590451 5683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b36c4279f6a484faa06c9f6c4cc06c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.608805 5773 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Bootstrap starting.
I20251028 14:52:29.609056 5772 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Bootstrap starting.
I20251028 14:52:29.609763 5771 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap starting.
I20251028 14:52:29.612393 5772 tablet_bootstrap.cc:654] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.613099 5771 tablet_bootstrap.cc:654] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.613125 5773 tablet_bootstrap.cc:654] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.614032 5772 log.cc:826] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:29.614404 5773 log.cc:826] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:29.614403 5771 log.cc:826] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:29.617738 5773 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: No bootstrap required, opened a new log
I20251028 14:52:29.617949 5771 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: No bootstrap required, opened a new log
I20251028 14:52:29.618098 5773 ts_tablet_manager.cc:1403] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20251028 14:52:29.618239 5771 ts_tablet_manager.cc:1403] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20251028 14:52:29.618326 5772 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: No bootstrap required, opened a new log
I20251028 14:52:29.618582 5772 ts_tablet_manager.cc:1403] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.001s
I20251028 14:52:29.624025 5773 raft_consensus.cc:359] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.624660 5773 raft_consensus.cc:385] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.624825 5773 raft_consensus.cc:740] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76b7e0f1dd442798012ff773075f578, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.625640 5773 consensus_queue.cc:260] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.625562 5772 raft_consensus.cc:359] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.625562 5771 raft_consensus.cc:359] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.625977 5771 raft_consensus.cc:385] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.625998 5772 raft_consensus.cc:385] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.626094 5772 raft_consensus.cc:740] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050795decf2492bbbefa78269460b5d, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.626116 5771 raft_consensus.cc:740] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.626745 5772 consensus_queue.cc:260] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.626745 5771 consensus_queue.cc:260] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.628505 5771 ts_tablet_manager.cc:1434] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20251028 14:52:29.628996 5771 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap starting.
I20251028 14:52:29.630990 5771 tablet_bootstrap.cc:654] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.633113 5618 heartbeater.cc:499] Master 127.0.104.62:38665 was elected leader, sending a full tablet report...
I20251028 14:52:29.633608 5773 ts_tablet_manager.cc:1434] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20251028 14:52:29.637231 5771 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: No bootstrap required, opened a new log
I20251028 14:52:29.637511 5771 ts_tablet_manager.cc:1403] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20251028 14:52:29.637962 5754 heartbeater.cc:499] Master 127.0.104.62:38665 was elected leader, sending a full tablet report...
I20251028 14:52:29.638530 5772 ts_tablet_manager.cc:1434] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Time spent starting tablet: real 0.020s user 0.009s sys 0.007s
I20251028 14:52:29.638445 5771 raft_consensus.cc:359] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.638626 5771 raft_consensus.cc:385] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.638742 5771 raft_consensus.cc:740] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.639019 5772 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Bootstrap starting.
I20251028 14:52:29.639010 5771 consensus_queue.cc:260] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.639516 5771 ts_tablet_manager.cc:1434] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 14:52:29.640753 5773 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Bootstrap starting.
I20251028 14:52:29.641445 5772 tablet_bootstrap.cc:654] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.642836 5773 tablet_bootstrap.cc:654] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.646440 5773 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: No bootstrap required, opened a new log
I20251028 14:52:29.646607 5773 ts_tablet_manager.cc:1403] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20251028 14:52:29.647298 5773 raft_consensus.cc:359] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.647459 5773 raft_consensus.cc:385] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.647544 5773 raft_consensus.cc:740] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76b7e0f1dd442798012ff773075f578, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.647787 5773 consensus_queue.cc:260] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.648756 5773 ts_tablet_manager.cc:1434] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:52:29.649626 5772 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: No bootstrap required, opened a new log
I20251028 14:52:29.649770 5772 ts_tablet_manager.cc:1403] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20251028 14:52:29.650439 5772 raft_consensus.cc:359] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.650595 5772 raft_consensus.cc:385] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.650684 5772 raft_consensus.cc:740] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050795decf2492bbbefa78269460b5d, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.650928 5772 consensus_queue.cc:260] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.651407 5772 ts_tablet_manager.cc:1434] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 14:52:29.702345 5779 raft_consensus.cc:493] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:29.702605 5779 raft_consensus.cc:515] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.704406 5779 leader_election.cc:290] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:29.711588 5437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" is_pre_election: true
I20251028 14:52:29.712333 5437 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 0.
I20251028 14:52:29.712625 5573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578" is_pre_election: true
I20251028 14:52:29.713146 5573 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 0.
I20251028 14:52:29.713425 5635 leader_election.cc:304] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d; no voters:
I20251028 14:52:29.713838 5779 raft_consensus.cc:2804] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:29.713973 5779 raft_consensus.cc:493] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:29.714069 5779 raft_consensus.cc:3060] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:29.716722 5779 raft_consensus.cc:515] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.717487 5779 leader_election.cc:290] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 election: Requested vote from peers 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:29.718437 5437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa"
I20251028 14:52:29.718437 5573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578"
I20251028 14:52:29.718684 5573 raft_consensus.cc:3060] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:29.718691 5437 raft_consensus.cc:3060] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:29.722141 5437 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 1.
I20251028 14:52:29.722139 5573 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 1.
I20251028 14:52:29.722695 5635 leader_election.cc:304] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d; no voters:
I20251028 14:52:29.723214 5779 raft_consensus.cc:2804] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:29.723731 5779 raft_consensus.cc:697] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 1 LEADER]: Becoming Leader. State: Replica: a050795decf2492bbbefa78269460b5d, State: Running, Role: LEADER
I20251028 14:52:29.724272 5779 consensus_queue.cc:237] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:29.729238 5277 catalog_manager.cc:5649] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d reported cstate change: term changed from 0 to 1, leader changed from <none> to a050795decf2492bbbefa78269460b5d (127.0.104.3). New cstate: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:29.741858 5483 tablet.cc:2378] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:29.746381 5619 tablet.cc:2378] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251028 14:52:29.758212 5755 tablet.cc:2378] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:52:29.798929 5779 raft_consensus.cc:493] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:29.799173 5779 raft_consensus.cc:515] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.799844 5779 leader_election.cc:290] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:29.800508 5573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578" is_pre_election: true
I20251028 14:52:29.800506 5437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" is_pre_election: true
I20251028 14:52:29.800765 5437 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 0.
I20251028 14:52:29.800781 5573 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 0.
I20251028 14:52:29.801335 5635 leader_election.cc:304] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:29.801704 5779 raft_consensus.cc:2804] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:29.801822 5779 raft_consensus.cc:493] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:29.801911 5779 raft_consensus.cc:3060] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:29.805058 5779 raft_consensus.cc:515] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.805905 5779 leader_election.cc:290] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 election: Requested vote from peers 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:29.806433 5437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa"
I20251028 14:52:29.806496 5573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578"
I20251028 14:52:29.806658 5437 raft_consensus.cc:3060] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:29.806713 5573 raft_consensus.cc:3060] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:29.808741 5437 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 1.
I20251028 14:52:29.808871 5573 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 1.
I20251028 14:52:29.809340 5635 leader_election.cc:304] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d; no voters:
I20251028 14:52:29.809738 5779 raft_consensus.cc:2804] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:29.809975 5779 raft_consensus.cc:697] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 1 LEADER]: Becoming Leader. State: Replica: a050795decf2492bbbefa78269460b5d, State: Running, Role: LEADER
I20251028 14:52:29.810288 5779 consensus_queue.cc:237] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.813226 5277 catalog_manager.cc:5649] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d reported cstate change: term changed from 0 to 1, leader changed from <none> to a050795decf2492bbbefa78269460b5d (127.0.104.3). New cstate: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:29.959614 5277 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49632:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:29.960733 5277 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:52:29.968303 5416 tablet_service.cc:1505] Processing CreateTablet for tablet 1e3bae4fdc174ae9ae5f4bdf97dd217c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54abbd888b954ca8865e68b5a77889f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:29.968493 5552 tablet_service.cc:1505] Processing CreateTablet for tablet 1e3bae4fdc174ae9ae5f4bdf97dd217c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54abbd888b954ca8865e68b5a77889f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:29.969391 5416 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3bae4fdc174ae9ae5f4bdf97dd217c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.968874 5683 tablet_service.cc:1505] Processing CreateTablet for tablet 1e3bae4fdc174ae9ae5f4bdf97dd217c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=54abbd888b954ca8865e68b5a77889f4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:29.969465 5552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3bae4fdc174ae9ae5f4bdf97dd217c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.969738 5683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e3bae4fdc174ae9ae5f4bdf97dd217c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:29.975847 5771 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap starting.
I20251028 14:52:29.977175 5773 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Bootstrap starting.
I20251028 14:52:29.977355 5772 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Bootstrap starting.
I20251028 14:52:29.977492 5771 tablet_bootstrap.cc:654] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.979089 5773 tablet_bootstrap.cc:654] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.979211 5772 tablet_bootstrap.cc:654] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:29.980237 5771 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: No bootstrap required, opened a new log
I20251028 14:52:29.980446 5771 ts_tablet_manager.cc:1403] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20251028 14:52:29.981144 5772 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: No bootstrap required, opened a new log
I20251028 14:52:29.981169 5773 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: No bootstrap required, opened a new log
I20251028 14:52:29.981340 5772 ts_tablet_manager.cc:1403] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251028 14:52:29.981489 5773 ts_tablet_manager.cc:1403] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20251028 14:52:29.981451 5771 raft_consensus.cc:359] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.981698 5771 raft_consensus.cc:385] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.981782 5771 raft_consensus.cc:740] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.982072 5772 raft_consensus.cc:359] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.982105 5771 consensus_queue.cc:260] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.982245 5772 raft_consensus.cc:385] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.982311 5772 raft_consensus.cc:740] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050795decf2492bbbefa78269460b5d, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.982298 5773 raft_consensus.cc:359] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.982501 5773 raft_consensus.cc:385] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:29.982568 5773 raft_consensus.cc:740] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76b7e0f1dd442798012ff773075f578, State: Initialized, Role: FOLLOWER
I20251028 14:52:29.982818 5773 consensus_queue.cc:260] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.982952 5778 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:29.983242 5778 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:29.983316 5773 ts_tablet_manager.cc:1434] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 14:52:29.983559 5777 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:29.982571 5772 consensus_queue.cc:260] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:29.983847 5777 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:29.984112 5779 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:29.984321 5779 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:29.984365 5771 ts_tablet_manager.cc:1434] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:29.985090 5772 ts_tablet_manager.cc:1434] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251028 14:52:29.999799 5779 raft_consensus.cc:493] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:52:30.000277 5779 raft_consensus.cc:515] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:30.001060 5779 leader_election.cc:290] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:30.001665 5437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" is_pre_election: true
I20251028 14:52:30.001675 5573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578" is_pre_election: true
I20251028 14:52:30.001914 5437 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 0.
I20251028 14:52:30.001914 5573 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 0.
I20251028 14:52:30.002429 5635 leader_election.cc:304] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:30.002766 5779 raft_consensus.cc:2804] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:52:30.002894 5779 raft_consensus.cc:493] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:52:30.002974 5779 raft_consensus.cc:3060] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:30.006083 5779 raft_consensus.cc:515] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:30.006860 5779 leader_election.cc:290] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 election: Requested vote from peers 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:30.007539 5573 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578"
I20251028 14:52:30.007558 5437 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "a050795decf2492bbbefa78269460b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa"
I20251028 14:52:30.007748 5437 raft_consensus.cc:3060] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:30.007742 5573 raft_consensus.cc:3060] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:30.010133 5573 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 1.
I20251028 14:52:30.010785 5637 leader_election.cc:304] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a050795decf2492bbbefa78269460b5d, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:30.010838 5437 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050795decf2492bbbefa78269460b5d in term 1.
I20251028 14:52:30.011430 5779 raft_consensus.cc:2804] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:30.011624 5779 raft_consensus.cc:697] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 1 LEADER]: Becoming Leader. State: Replica: a050795decf2492bbbefa78269460b5d, State: Running, Role: LEADER
I20251028 14:52:30.012079 5779 consensus_queue.cc:237] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:30.014142 5784 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: TxnStatusTablet state changed. Reason: New leader a050795decf2492bbbefa78269460b5d. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.014392 5784 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:30.015118 5798 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:30.015447 5798 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:30.016180 5277 catalog_manager.cc:5649] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d reported cstate change: term changed from 0 to 1, leader changed from <none> to a050795decf2492bbbefa78269460b5d (127.0.104.3). New cstate: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:30.047802 5437 raft_consensus.cc:1275] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Refusing update from remote peer a050795decf2492bbbefa78269460b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:30.047811 5573 raft_consensus.cc:1275] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Refusing update from remote peer a050795decf2492bbbefa78269460b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:30.049134 5784 consensus_queue.cc:1048] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:30.049583 5779 consensus_queue.cc:1048] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:30.058292 5778 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: TxnStatusTablet state changed. Reason: New leader a050795decf2492bbbefa78269460b5d. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.058542 5778 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:30.059880 5801 mvcc.cc:204] Tried to move back new op lower bound from 7215772262584438784 to 7215772262451965952. Current Snapshot: MvccSnapshot[applied={T|T < 7215772262584438784}]
I20251028 14:52:30.060537 5802 mvcc.cc:204] Tried to move back new op lower bound from 7215772262584438784 to 7215772262451965952. Current Snapshot: MvccSnapshot[applied={T|T < 7215772262584438784}]
I20251028 14:52:30.061349 5778 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.061546 5778 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:30.064760 5779 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.065064 5779 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:30.067358 5777 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: TxnStatusTablet state changed. Reason: New leader a050795decf2492bbbefa78269460b5d. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.067652 5777 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:30.069907 5784 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.070508 5784 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:30.090066 5777 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.090328 5777 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:30.128175 5573 raft_consensus.cc:1275] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Refusing update from remote peer a050795decf2492bbbefa78269460b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:30.128840 5815 consensus_queue.cc:1048] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:30.129357 5437 raft_consensus.cc:1275] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Refusing update from remote peer a050795decf2492bbbefa78269460b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:52:30.130003 5815 consensus_queue.cc:1048] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:30.138089 5803 mvcc.cc:204] Tried to move back new op lower bound from 7215772262918852608 to 7215772261275238400. Current Snapshot: MvccSnapshot[applied={T|T < 7215772262918852608}]
W20251028 14:52:30.203776 5812 tablet_replica.cc:1306] Illegal state: Failed to write to server: a050795decf2492bbbefa78269460b5d (127.0.104.3:33277): transaction ID 0 not open: COMMITTED: cancelling pending write operations
W20251028 14:52:30.204057 5812 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:36822: Illegal state: Failed to write to server: a050795decf2492bbbefa78269460b5d (127.0.104.3:33277): transaction ID 0 not open: COMMITTED
W20251028 14:52:30.204824 5757 batcher.cc:441] Illegal state: Failed to write batch of 1 ops to tablet 0b36c4279f6a484faa06c9f6c4cc06c2 after 1 attempt(s): Failed to write to server: a050795decf2492bbbefa78269460b5d (127.0.104.3:33277): Failed to write to server: a050795decf2492bbbefa78269460b5d (127.0.104.3:33277): transaction ID 0 not open: COMMITTED
I20251028 14:52:30.273026 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 5347
I20251028 14:52:30.274353 5477 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:30.334185 5784 consensus_queue.cc:1048] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:30.389387 5815 consensus_queue.cc:1048] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:52:30.390131 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 5347
W20251028 14:52:30.405879 5635 connection.cc:537] client connection to 127.0.104.1:41875 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251028 14:52:30.406191 5635 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:52:30.406749 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:41875
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:38665
--webserver_port=44467
--webserver_interface=127.0.104.1
--builtin_ntp_servers=127.0.104.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:52:30.411494 5635 consensus_peers.cc:597] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d -> Peer 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875): Couldn't send request to peer 852e2c5586bc46ca81ae46d74d4ed7fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:41875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251028 14:52:30.665351 5635 consensus_peers.cc:597] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d -> Peer 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875): Couldn't send request to peer 852e2c5586bc46ca81ae46d74d4ed7fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:41875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251028 14:52:30.772032 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:30.772390 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:30.772511 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:30.780980 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:30.781162 5823 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:30.781319 5823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251028 14:52:30.793234 5823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:41875
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=44467
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:38665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:30.795070 5823 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:30.797344 5823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:30.808825 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:30.808785 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:30.809584 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:30.811121 5823 server_base.cc:1047] running on GCE node
I20251028 14:52:30.811851 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:30.813192 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:30.814626 5823 hybrid_clock.cc:648] HybridClock initialized: now 1761663150814536 us; error 92 us; skew 500 ppm
I20251028 14:52:30.815034 5823 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:30.818110 5823 webserver.cc:492] Webserver started at http://127.0.104.1:44467/ using document root <none> and password file <none>
I20251028 14:52:30.819015 5823 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:30.819164 5823 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:30.825706 5823 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:30.829373 5837 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:30.831379 5823 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20251028 14:52:30.831581 5823 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
uuid: "852e2c5586bc46ca81ae46d74d4ed7fa"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:30.832369 5823 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251028 14:52:30.835317 5635 consensus_peers.cc:597] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d -> Peer 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875): Couldn't send request to peer 852e2c5586bc46ca81ae46d74d4ed7fa. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:41875: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251028 14:52:30.871263 5823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:30.872215 5823 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:30.872473 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:30.877372 5845 ts_tablet_manager.cc:548] Loading tablet metadata (0/3 complete)
I20251028 14:52:30.897887 5823 ts_tablet_manager.cc:585] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20251028 14:52:30.898043 5823 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.021s user 0.001s sys 0.000s
I20251028 14:52:30.898219 5823 ts_tablet_manager.cc:600] Registering tablets (0/3 complete)
I20251028 14:52:30.905323 5845 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap starting.
I20251028 14:52:30.910957 5823 ts_tablet_manager.cc:616] Registered 3 tablets
I20251028 14:52:30.911124 5823 ts_tablet_manager.cc:595] Time spent register tablets: real 0.013s user 0.011s sys 0.000s
I20251028 14:52:30.927451 5845 log.cc:826] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:30.931345 5845 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:30.932179 5845 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap complete.
I20251028 14:52:30.933415 5845 ts_tablet_manager.cc:1403] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20251028 14:52:30.941102 5845 raft_consensus.cc:359] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:30.942893 5845 raft_consensus.cc:740] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Initialized, Role: FOLLOWER
I20251028 14:52:30.943987 5845 consensus_queue.cc:260] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:30.946811 5845 ts_tablet_manager.cc:1434] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20251028 14:52:30.947769 5845 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap starting.
I20251028 14:52:30.971248 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:41875
I20251028 14:52:30.971292 5961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:41875 every 8 connection(s)
I20251028 14:52:30.973416 5823 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-0/data/info.pb
W20251028 14:52:30.976472 5963 tablet.cc:2378] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:52:30.976743 5962 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38665
I20251028 14:52:30.977150 5962 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:30.978184 5962 heartbeater.cc:507] Master 127.0.104.62:38665 requested a full tablet report, sending...
I20251028 14:52:30.981674 5277 ts_manager.cc:194] Re-registered known tserver with Master: 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875)
I20251028 14:52:30.982196 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 5823
I20251028 14:52:30.982481 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 5484
I20251028 14:52:30.983614 5613 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:30.983873 5277 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:49251
I20251028 14:52:30.986195 5845 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=4 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:30.986742 5845 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap complete.
I20251028 14:52:30.987543 5845 ts_tablet_manager.cc:1403] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent bootstrapping tablet: real 0.040s user 0.021s sys 0.016s
I20251028 14:52:30.988471 5845 raft_consensus.cc:359] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:30.988684 5845 raft_consensus.cc:740] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Initialized, Role: FOLLOWER
I20251028 14:52:30.989032 5845 consensus_queue.cc:260] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:30.989825 5962 heartbeater.cc:499] Master 127.0.104.62:38665 was elected leader, sending a full tablet report...
I20251028 14:52:30.989753 5904 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:30.989943 5845 ts_tablet_manager.cc:1434] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:30.990094 5904 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:30.990532 5845 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap starting.
I20251028 14:52:31.020123 5845 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:31.020761 5845 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Bootstrap complete.
I20251028 14:52:31.022013 5845 ts_tablet_manager.cc:1403] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.011s
I20251028 14:52:31.022886 5845 raft_consensus.cc:359] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:31.023234 5845 raft_consensus.cc:740] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Initialized, Role: FOLLOWER
I20251028 14:52:31.023572 5845 consensus_queue.cc:260] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:31.024173 5845 ts_tablet_manager.cc:1434] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251028 14:52:31.090945 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 5484
I20251028 14:52:31.107038 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:38183
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:38665
--webserver_port=44665
--webserver_interface=127.0.104.2
--builtin_ntp_servers=127.0.104.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:52:31.216306 5637 consensus_peers.cc:597] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d -> Peer b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183): Couldn't send request to peer b76b7e0f1dd442798012ff773075f578. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.2:38183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251028 14:52:31.245359 5637 consensus_peers.cc:597] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d -> Peer b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183): Couldn't send request to peer b76b7e0f1dd442798012ff773075f578. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.2:38183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251028 14:52:31.249641 5815 tablet_replica.cc:442] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:31.250036 5815 tablet_replica.cc:445] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:31.270036 5904 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader a050795decf2492bbbefa78269460b5d. Latest consensus state: current_term: 1 leader_uuid: "a050795decf2492bbbefa78269460b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:31.270341 5904 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: FOLLOWER
W20251028 14:52:31.441183 5637 consensus_peers.cc:597] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d -> Peer b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183): Couldn't send request to peer b76b7e0f1dd442798012ff773075f578. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.2:38183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251028 14:52:31.473433 5966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:31.473791 5966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:31.473927 5966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:31.483479 5966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:31.483667 5966 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:31.483798 5966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251028 14:52:31.494971 5966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:38183
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=44665
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:38665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:31.496693 5966 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:31.498661 5966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:31.508841 5977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:31.509447 5978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:31.509984 5980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:31.511656 5966 server_base.cc:1047] running on GCE node
I20251028 14:52:31.512426 5966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:31.513604 5966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:31.514858 5966 hybrid_clock.cc:648] HybridClock initialized: now 1761663151514775 us; error 69 us; skew 500 ppm
I20251028 14:52:31.515204 5966 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:31.517882 5966 webserver.cc:492] Webserver started at http://127.0.104.2:44665/ using document root <none> and password file <none>
I20251028 14:52:31.518743 5966 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:31.518895 5966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:31.525349 5966 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20251028 14:52:31.528764 5986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:31.530397 5966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20251028 14:52:31.530566 5966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
uuid: "b76b7e0f1dd442798012ff773075f578"
format_stamp: "Formatted at 2025-10-28 14:52:28 on dist-test-slave-0v9m"
I20251028 14:52:31.531327 5966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:31.545339 5966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:31.546231 5966 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:31.546489 5966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:31.550642 5994 ts_tablet_manager.cc:548] Loading tablet metadata (0/3 complete)
I20251028 14:52:31.564321 5966 ts_tablet_manager.cc:585] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20251028 14:52:31.564481 5966 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.001s
I20251028 14:52:31.564632 5966 ts_tablet_manager.cc:600] Registering tablets (0/3 complete)
I20251028 14:52:31.569614 5994 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Bootstrap starting.
I20251028 14:52:31.575345 5966 ts_tablet_manager.cc:616] Registered 3 tablets
I20251028 14:52:31.575538 5966 ts_tablet_manager.cc:595] Time spent register tablets: real 0.011s user 0.002s sys 0.010s
I20251028 14:52:31.590157 5994 log.cc:826] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:31.596767 5994 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:31.597498 5994 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Bootstrap complete.
I20251028 14:52:31.599113 5994 ts_tablet_manager.cc:1403] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20251028 14:52:31.607666 5994 raft_consensus.cc:359] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:31.608769 5994 raft_consensus.cc:740] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76b7e0f1dd442798012ff773075f578, State: Initialized, Role: FOLLOWER
I20251028 14:52:31.609612 5994 consensus_queue.cc:260] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:31.611460 5994 ts_tablet_manager.cc:1434] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20251028 14:52:31.611897 5994 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Bootstrap starting.
I20251028 14:52:31.627800 5966 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38183
I20251028 14:52:31.627846 6109 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38183 every 8 connection(s)
I20251028 14:52:31.629472 5966 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-1/data/info.pb
I20251028 14:52:31.630550 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 5966
I20251028 14:52:31.631265 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 5620
I20251028 14:52:31.633236 5749 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:31.633764 6111 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38665
I20251028 14:52:31.634192 6111 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:31.635411 6111 heartbeater.cc:507] Master 127.0.104.62:38665 requested a full tablet report, sending...
I20251028 14:52:31.639433 5276 ts_manager.cc:194] Re-registered known tserver with Master: b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:31.641784 5276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:44243
I20251028 14:52:31.644577 5994 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=4 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:31.645049 5994 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Bootstrap complete.
I20251028 14:52:31.645864 5994 ts_tablet_manager.cc:1403] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Time spent bootstrapping tablet: real 0.034s user 0.013s sys 0.019s
I20251028 14:52:31.646586 5994 raft_consensus.cc:359] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:31.646749 5994 raft_consensus.cc:740] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76b7e0f1dd442798012ff773075f578, State: Initialized, Role: FOLLOWER
I20251028 14:52:31.647020 5994 consensus_queue.cc:260] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:31.647580 6111 heartbeater.cc:499] Master 127.0.104.62:38665 was elected leader, sending a full tablet report...
I20251028 14:52:31.647480 6077 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:31.647871 6077 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:31.647619 5994 ts_tablet_manager.cc:1434] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 14:52:31.648274 5994 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Bootstrap starting.
I20251028 14:52:31.670670 5994 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:31.671216 5994 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Bootstrap complete.
I20251028 14:52:31.671988 5994 ts_tablet_manager.cc:1403] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.008s
I20251028 14:52:31.672705 5994 raft_consensus.cc:359] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:31.672863 5994 raft_consensus.cc:740] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76b7e0f1dd442798012ff773075f578, State: Initialized, Role: FOLLOWER
I20251028 14:52:31.673125 5994 consensus_queue.cc:260] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:31.673658 5994 ts_tablet_manager.cc:1434] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20251028 14:52:31.767737 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 5620
W20251028 14:52:31.788575 5263 connection.cc:537] server connection from 127.0.104.3:57527 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251028 14:52:31.789978 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
/tmp/dist-test-taskCXPJAB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:33277
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:38665
--webserver_port=46041
--webserver_interface=127.0.104.3
--builtin_ntp_servers=127.0.104.20:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20251028 14:52:31.790477 6115 negotiation.cc:337] Failed RPC negotiation. Trace:
1028 14:52:31.767597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:45155 (local address 127.0.104.2:38183)
1028 14:52:31.769041 (+ 1444us) server_negotiation.cc:207] Beginning negotiation
1028 14:52:31.769053 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1028 14:52:31.769102 (+ 49us) server_negotiation.cc:408] Connection header received
1028 14:52:31.769294 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1028 14:52:31.769304 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1028 14:52:31.769674 (+ 370us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1028 14:52:31.769954 (+ 280us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1028 14:52:31.788717 (+ 18763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:45155: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":1209,"thread_start_us":3810,"threads_started":1}
W20251028 14:52:32.144477 6116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251028 14:52:32.144812 6116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251028 14:52:32.144905 6116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251028 14:52:32.153299 6116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251028 14:52:32.153462 6116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251028 14:52:32.153543 6116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251028 14:52:32.163733 6116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:33277
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=46041
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:38665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 28 Oct 2025 14:43:23 UTC on 24a791456cd2
build id 8760
ASAN enabled
I20251028 14:52:32.165436 6116 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:32.167250 6116 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:32.177879 6121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:32.177893 6122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:32.178598 6124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:32.178990 6116 server_base.cc:1047] running on GCE node
I20251028 14:52:32.179602 6116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251028 14:52:32.180610 6116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251028 14:52:32.181840 6116 hybrid_clock.cc:648] HybridClock initialized: now 1761663152181749 us; error 72 us; skew 500 ppm
I20251028 14:52:32.182166 6116 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:32.184499 6116 webserver.cc:492] Webserver started at http://127.0.104.3:46041/ using document root <none> and password file <none>
I20251028 14:52:32.185313 6116 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:32.185464 6116 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:32.191124 6116 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20251028 14:52:32.194649 6130 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:32.196223 6116 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251028 14:52:32.196409 6116 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
uuid: "a050795decf2492bbbefa78269460b5d"
format_stamp: "Formatted at 2025-10-28 14:52:29 on dist-test-slave-0v9m"
I20251028 14:52:32.197203 6116 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:32.216742 6116 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:32.217725 6116 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:32.218014 6116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:32.222139 6138 ts_tablet_manager.cc:548] Loading tablet metadata (0/3 complete)
I20251028 14:52:32.236191 6116 ts_tablet_manager.cc:585] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20251028 14:52:32.236569 6116 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20251028 14:52:32.236799 6116 ts_tablet_manager.cc:600] Registering tablets (0/3 complete)
I20251028 14:52:32.242056 6138 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Bootstrap starting.
I20251028 14:52:32.247152 6116 ts_tablet_manager.cc:616] Registered 3 tablets
I20251028 14:52:32.247299 6116 ts_tablet_manager.cc:595] Time spent register tablets: real 0.011s user 0.002s sys 0.006s
I20251028 14:52:32.263892 6138 log.cc:826] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:32.269943 6138 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:32.270840 6138 tablet_bootstrap.cc:492] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Bootstrap complete.
I20251028 14:52:32.272399 6138 ts_tablet_manager.cc:1403] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.014s
I20251028 14:52:32.280287 6138 raft_consensus.cc:359] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.281734 6138 raft_consensus.cc:740] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050795decf2492bbbefa78269460b5d, State: Initialized, Role: FOLLOWER
I20251028 14:52:32.282553 6138 consensus_queue.cc:260] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.284180 6138 ts_tablet_manager.cc:1434] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20251028 14:52:32.284606 6138 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Bootstrap starting.
I20251028 14:52:32.294486 6116 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:33277
I20251028 14:52:32.294504 6253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:33277 every 8 connection(s)
I20251028 14:52:32.296152 6116 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1761663088357144-416-0/minicluster-data/ts-2/data/info.pb
I20251028 14:52:32.300048 6254 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38665
I20251028 14:52:32.300523 6254 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:32.301777 6254 heartbeater.cc:507] Master 127.0.104.62:38665 requested a full tablet report, sending...
I20251028 14:52:32.303596 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu as pid 6116
I20251028 14:52:32.306943 5276 ts_manager.cc:194] Re-registered known tserver with Master: a050795decf2492bbbefa78269460b5d (127.0.104.3:33277)
I20251028 14:52:32.308988 5276 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:52231
I20251028 14:52:32.318110 6138 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=4 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:32.318604 6138 tablet_bootstrap.cc:492] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Bootstrap complete.
I20251028 14:52:32.319440 6138 ts_tablet_manager.cc:1403] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Time spent bootstrapping tablet: real 0.035s user 0.024s sys 0.008s
I20251028 14:52:32.320220 6138 raft_consensus.cc:359] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.320401 6138 raft_consensus.cc:740] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050795decf2492bbbefa78269460b5d, State: Initialized, Role: FOLLOWER
I20251028 14:52:32.320816 6138 consensus_queue.cc:260] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.321467 6254 heartbeater.cc:499] Master 127.0.104.62:38665 was elected leader, sending a full tablet report...
I20251028 14:52:32.321516 6138 ts_tablet_manager.cc:1434] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20251028 14:52:32.321467 6236 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:32.321854 6236 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:52:32.321907 6138 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Bootstrap starting.
I20251028 14:52:32.344235 6138 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:32.344681 6138 tablet_bootstrap.cc:492] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Bootstrap complete.
I20251028 14:52:32.345502 6138 ts_tablet_manager.cc:1403] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.011s
I20251028 14:52:32.346165 6138 raft_consensus.cc:359] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:32.346320 6138 raft_consensus.cc:740] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050795decf2492bbbefa78269460b5d, State: Initialized, Role: FOLLOWER
I20251028 14:52:32.346560 6138 consensus_queue.cc:260] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:32.346992 6138 ts_tablet_manager.cc:1434] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251028 14:52:32.771243 6266 raft_consensus.cc:493] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a050795decf2492bbbefa78269460b5d)
I20251028 14:52:32.771514 6266 raft_consensus.cc:515] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.773197 6266 leader_election.cc:290] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183), a050795decf2492bbbefa78269460b5d (127.0.104.3:33277)
I20251028 14:52:32.781194 6208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a050795decf2492bbbefa78269460b5d" is_pre_election: true
I20251028 14:52:32.781195 6064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578" is_pre_election: true
I20251028 14:52:32.781674 6064 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 1.
I20251028 14:52:32.781672 6208 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 1.
I20251028 14:52:32.782262 5840 leader_election.cc:304] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:32.782634 6266 raft_consensus.cc:2804] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251028 14:52:32.782765 6266 raft_consensus.cc:493] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader a050795decf2492bbbefa78269460b5d)
I20251028 14:52:32.782862 6266 raft_consensus.cc:3060] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.786537 6266 raft_consensus.cc:515] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.787248 6266 leader_election.cc:290] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 election: Requested vote from peers b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183), a050795decf2492bbbefa78269460b5d (127.0.104.3:33277)
I20251028 14:52:32.787767 6208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a050795decf2492bbbefa78269460b5d"
I20251028 14:52:32.787771 6064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1e3bae4fdc174ae9ae5f4bdf97dd217c" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578"
I20251028 14:52:32.787961 6064 raft_consensus.cc:3060] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.787964 6208 raft_consensus.cc:3060] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.791714 6208 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P a050795decf2492bbbefa78269460b5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 2.
I20251028 14:52:32.791714 6064 raft_consensus.cc:2468] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P b76b7e0f1dd442798012ff773075f578 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 2.
I20251028 14:52:32.792276 5839 leader_election.cc:304] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:32.792660 6266 raft_consensus.cc:2804] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 FOLLOWER]: Leader election won for term 2
I20251028 14:52:32.793190 6266 raft_consensus.cc:697] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 LEADER]: Becoming Leader. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Running, Role: LEADER
I20251028 14:52:32.793695 6266 consensus_queue.cc:237] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.795620 6269 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader 852e2c5586bc46ca81ae46d74d4ed7fa. Latest consensus state: current_term: 2 leader_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } }
I20251028 14:52:32.795997 6269 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:32.796959 6271 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:32.797220 6271 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:32.797825 5276 catalog_manager.cc:5649] T 1e3bae4fdc174ae9ae5f4bdf97dd217c P 852e2c5586bc46ca81ae46d74d4ed7fa reported cstate change: term changed from 1 to 2, leader changed from a050795decf2492bbbefa78269460b5d (127.0.104.3) to 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1). New cstate: current_term: 2 leader_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:32.801863 6269 raft_consensus.cc:493] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a050795decf2492bbbefa78269460b5d)
I20251028 14:52:32.802073 6269 raft_consensus.cc:515] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:32.802995 6269 leader_election.cc:290] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a050795decf2492bbbefa78269460b5d (127.0.104.3:33277), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:32.803793 6064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578" is_pre_election: true
I20251028 14:52:32.803788 6208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a050795decf2492bbbefa78269460b5d" is_pre_election: true
I20251028 14:52:32.804056 6064 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 1.
I20251028 14:52:32.804073 6208 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 1.
I20251028 14:52:32.804589 5839 leader_election.cc:304] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:32.805163 6269 raft_consensus.cc:2804] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251028 14:52:32.805327 6269 raft_consensus.cc:493] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader a050795decf2492bbbefa78269460b5d)
I20251028 14:52:32.805428 6269 raft_consensus.cc:3060] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.805977 6266 raft_consensus.cc:493] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a050795decf2492bbbefa78269460b5d)
I20251028 14:52:32.806167 6266 raft_consensus.cc:515] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.806864 6266 leader_election.cc:290] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183), a050795decf2492bbbefa78269460b5d (127.0.104.3:33277)
I20251028 14:52:32.807401 6064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578" is_pre_election: true
I20251028 14:52:32.807646 6064 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 1.
I20251028 14:52:32.807618 6208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a050795decf2492bbbefa78269460b5d" is_pre_election: true
I20251028 14:52:32.807845 6208 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 1.
I20251028 14:52:32.808259 5840 leader_election.cc:304] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:32.808724 6266 raft_consensus.cc:2804] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251028 14:52:32.808871 6266 raft_consensus.cc:493] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Starting leader election (detected failure of leader a050795decf2492bbbefa78269460b5d)
I20251028 14:52:32.809012 6266 raft_consensus.cc:3060] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.808879 6269 raft_consensus.cc:515] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:32.810067 6208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a050795decf2492bbbefa78269460b5d"
I20251028 14:52:32.810078 6064 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa2fc80f5ab94e9699132e99a180de02" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578"
I20251028 14:52:32.810298 6064 raft_consensus.cc:3060] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.810298 6208 raft_consensus.cc:3060] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.812206 6266 raft_consensus.cc:515] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.813591 6063 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b76b7e0f1dd442798012ff773075f578"
I20251028 14:52:32.813740 6064 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P b76b7e0f1dd442798012ff773075f578 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 2.
I20251028 14:52:32.813936 6063 raft_consensus.cc:3060] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.814127 6269 leader_election.cc:290] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 election: Requested vote from peers a050795decf2492bbbefa78269460b5d (127.0.104.3:33277), b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183)
I20251028 14:52:32.814373 6208 raft_consensus.cc:2468] T aa2fc80f5ab94e9699132e99a180de02 P a050795decf2492bbbefa78269460b5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 2.
I20251028 14:52:32.814987 5839 leader_election.cc:304] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, a050795decf2492bbbefa78269460b5d; no voters:
I20251028 14:52:32.815328 6269 raft_consensus.cc:2804] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 FOLLOWER]: Leader election won for term 2
I20251028 14:52:32.815740 6269 raft_consensus.cc:697] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 LEADER]: Becoming Leader. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Running, Role: LEADER
I20251028 14:52:32.816260 6269 consensus_queue.cc:237] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } }
I20251028 14:52:32.817211 6063 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P b76b7e0f1dd442798012ff773075f578 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 2.
I20251028 14:52:32.818177 5840 leader_election.cc:304] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 852e2c5586bc46ca81ae46d74d4ed7fa, b76b7e0f1dd442798012ff773075f578; no voters:
I20251028 14:52:32.818460 6277 raft_consensus.cc:2804] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 FOLLOWER]: Leader election won for term 2
I20251028 14:52:32.818707 6266 leader_election.cc:290] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [CANDIDATE]: Term 2 election: Requested vote from peers b76b7e0f1dd442798012ff773075f578 (127.0.104.2:38183), a050795decf2492bbbefa78269460b5d (127.0.104.3:33277)
I20251028 14:52:32.819475 5276 catalog_manager.cc:5649] T aa2fc80f5ab94e9699132e99a180de02 P 852e2c5586bc46ca81ae46d74d4ed7fa reported cstate change: term changed from 1 to 2, leader changed from a050795decf2492bbbefa78269460b5d (127.0.104.3) to 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1). New cstate: current_term: 2 leader_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } health_report { overall_health: UNKNOWN } } }
I20251028 14:52:32.821848 6208 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b36c4279f6a484faa06c9f6c4cc06c2" candidate_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a050795decf2492bbbefa78269460b5d"
I20251028 14:52:32.822082 6208 raft_consensus.cc:3060] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:32.824200 6208 raft_consensus.cc:2468] T 0b36c4279f6a484faa06c9f6c4cc06c2 P a050795decf2492bbbefa78269460b5d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 852e2c5586bc46ca81ae46d74d4ed7fa in term 2.
I20251028 14:52:32.825692 6277 raft_consensus.cc:697] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [term 2 LEADER]: Becoming Leader. State: Replica: 852e2c5586bc46ca81ae46d74d4ed7fa, State: Running, Role: LEADER
I20251028 14:52:32.826143 6277 consensus_queue.cc:237] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } }
I20251028 14:52:32.830003 5276 catalog_manager.cc:5649] T 0b36c4279f6a484faa06c9f6c4cc06c2 P 852e2c5586bc46ca81ae46d74d4ed7fa reported cstate change: term changed from 1 to 2, leader changed from a050795decf2492bbbefa78269460b5d (127.0.104.3) to 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1). New cstate: current_term: 2 leader_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "852e2c5586bc46ca81ae46d74d4ed7fa" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b76b7e0f1dd442798012ff773075f578" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050795decf2492bbbefa78269460b5d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33277 } health_report { overall_health: UNKNOWN } } }
W20251028 14:52:32.862268 6262 tablet_replica.cc:1306] Illegal state: Failed to write to server: 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875): transaction ID 1 not open: COMMITTED: cancelling pending write operations
W20251028 14:52:32.862623 6262 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34498: Illegal state: Failed to write to server: 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875): transaction ID 1 not open: COMMITTED
W20251028 14:52:32.863677 5756 batcher.cc:441] Illegal state: Failed to write batch of 1 ops to tablet aa2fc80f5ab94e9699132e99a180de02 after 2 attempt(s): Failed to write to server: 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875): Failed to write to server: 852e2c5586bc46ca81ae46d74d4ed7fa (127.0.104.1:41875): transaction ID 1 not open: COMMITTED
I20251028 14:52:32.864858 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 5823
I20251028 14:52:32.874712 5957 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:32.984951 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 5823
I20251028 14:52:33.001884 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 5966
I20251028 14:52:33.010393 6105 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:33.107221 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 5966
I20251028 14:52:33.124037 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 6116
I20251028 14:52:33.131680 6249 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:33.224519 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 6116
I20251028 14:52:33.239315 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu pid 5247
I20251028 14:52:33.240620 5318 generic_service.cc:196] Checking for leaks (request via RPC)
I20251028 14:52:33.370060 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskCXPJAB/build/asan/bin/kudu with pid 5247
2025-10-28T14:52:33Z chronyd exiting
[ OK ] TxnWriteOpsITest.TxnWriteAfterCommit (6037 ms)
[----------] 8 tests from TxnWriteOpsITest (64982 ms total)
[----------] 18 tests from TxnOpDispatcherITest
[ RUN ] TxnOpDispatcherITest.LifecycleBasic
I20251028 14:52:33.417169 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:41081
I20251028 14:52:33.418499 416 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251028 14:52:33.419481 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:33.427052 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:33.427081 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:33.428542 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:33.429003 416 server_base.cc:1047] running on GCE node
I20251028 14:52:33.429891 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:33.430071 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:33.430203 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663153430163 us; error 0 us; skew 500 ppm
I20251028 14:52:33.430603 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:33.433054 416 webserver.cc:492] Webserver started at http://127.0.104.62:46825/ using document root <none> and password file <none>
I20251028 14:52:33.434226 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:33.434360 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:33.434688 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:33.437661 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "52bbb805d93340e58f5b5309c0f82aed"
format_stamp: "Formatted at 2025-10-28 14:52:33 on dist-test-slave-0v9m"
I20251028 14:52:33.444917 416 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20251028 14:52:33.449519 6295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:33.451557 416 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:33.451900 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "52bbb805d93340e58f5b5309c0f82aed"
format_stamp: "Formatted at 2025-10-28 14:52:33 on dist-test-slave-0v9m"
I20251028 14:52:33.452304 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:33.470356 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:33.471397 416 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251028 14:52:33.471676 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:33.496850 6356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41081 every 8 connection(s)
I20251028 14:52:33.496872 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41081
I20251028 14:52:33.500411 6357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:33.511722 6357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed: Bootstrap starting.
I20251028 14:52:33.515128 6357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:33.516364 6357 log.cc:826] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed: Log is configured to *not* fsync() on all Append() calls
I20251028 14:52:33.519266 6357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed: No bootstrap required, opened a new log
I20251028 14:52:33.523733 6357 raft_consensus.cc:359] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbb805d93340e58f5b5309c0f82aed" member_type: VOTER }
I20251028 14:52:33.524098 6357 raft_consensus.cc:385] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:33.524183 6357 raft_consensus.cc:740] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52bbb805d93340e58f5b5309c0f82aed, State: Initialized, Role: FOLLOWER
I20251028 14:52:33.524762 6357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbb805d93340e58f5b5309c0f82aed" member_type: VOTER }
I20251028 14:52:33.524962 6357 raft_consensus.cc:399] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:33.525065 6357 raft_consensus.cc:493] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:33.525204 6357 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:33.527460 6357 raft_consensus.cc:515] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbb805d93340e58f5b5309c0f82aed" member_type: VOTER }
I20251028 14:52:33.528003 6357 leader_election.cc:304] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52bbb805d93340e58f5b5309c0f82aed; no voters:
I20251028 14:52:33.528699 6357 leader_election.cc:290] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:33.528841 6360 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:33.529568 6360 raft_consensus.cc:697] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 1 LEADER]: Becoming Leader. State: Replica: 52bbb805d93340e58f5b5309c0f82aed, State: Running, Role: LEADER
I20251028 14:52:33.530158 6360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbb805d93340e58f5b5309c0f82aed" member_type: VOTER }
I20251028 14:52:33.530936 6357 sys_catalog.cc:565] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:33.533612 6362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52bbb805d93340e58f5b5309c0f82aed. Latest consensus state: current_term: 1 leader_uuid: "52bbb805d93340e58f5b5309c0f82aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbb805d93340e58f5b5309c0f82aed" member_type: VOTER } }
I20251028 14:52:33.533921 6362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:33.533974 6361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52bbb805d93340e58f5b5309c0f82aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52bbb805d93340e58f5b5309c0f82aed" member_type: VOTER } }
I20251028 14:52:33.534250 6361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:33.535848 6369 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:33.541850 6369 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:33.542624 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:52:33.556010 6369 catalog_manager.cc:1357] Generated new cluster ID: c696de1814374ec982d5bae5f0b123a8
I20251028 14:52:33.556160 6369 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:33.578336 6369 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:33.579747 6369 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:33.592293 6369 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed: Generated new TSK 0
I20251028 14:52:33.593312 6369 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:33.610689 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251028 14:52:33.609418 6312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46408:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:52:33.616266 6385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:33.616726 6386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:33.618031 6357 master.cc:464] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 1; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
I20251028 14:52:33.618889 416 server_base.cc:1047] running on GCE node
W20251028 14:52:33.619119 6388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:33.619542 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:33.619616 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:33.619692 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663153619654 us; error 0 us; skew 500 ppm
I20251028 14:52:33.619865 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:33.621364 416 webserver.cc:492] Webserver started at http://127.0.104.1:46769/ using document root <none> and password file <none>
I20251028 14:52:33.621860 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:33.622159 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:33.622363 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:33.623150 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "aa6e55c6194c420086dc337d79f7cf5f"
format_stamp: "Formatted at 2025-10-28 14:52:33 on dist-test-slave-0v9m"
I20251028 14:52:33.626616 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251028 14:52:33.629861 6393 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:33.630395 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:33.630542 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "aa6e55c6194c420086dc337d79f7cf5f"
format_stamp: "Formatted at 2025-10-28 14:52:33 on dist-test-slave-0v9m"
I20251028 14:52:33.630707 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:33.656728 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:33.657938 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:33.662395 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:33.662591 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:33.662762 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:33.662874 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:33.692971 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45933
I20251028 14:52:33.692991 6462 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45933 every 8 connection(s)
I20251028 14:52:33.695592 6463 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41081
I20251028 14:52:33.695894 6463 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:33.696686 6463 heartbeater.cc:507] Master 127.0.104.62:41081 requested a full tablet report, sending...
I20251028 14:52:33.698659 6312 ts_manager.cc:194] Registered new tserver with Master: aa6e55c6194c420086dc337d79f7cf5f (127.0.104.1:45933)
I20251028 14:52:33.699129 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004687749s
I20251028 14:52:33.700932 6312 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46418
I20251028 14:52:33.714277 6312 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46432:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:52:33.738941 6427 tablet_service.cc:1505] Processing CreateTablet for tablet 1a1c45be1a874c8c8cb2b93ec3ab4b1d (DEFAULT_TABLE table=txn_op_dispatcher_test [id=3b46eccbdcad4e5b9ad4b7497183cf02]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:33.738900 6426 tablet_service.cc:1505] Processing CreateTablet for tablet 525ffdcfed074308943cbcc791fe6d85 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=3b46eccbdcad4e5b9ad4b7497183cf02]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:33.739907 6426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 525ffdcfed074308943cbcc791fe6d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:33.740450 6427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a1c45be1a874c8c8cb2b93ec3ab4b1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:33.750751 6473 tablet_bootstrap.cc:492] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f: Bootstrap starting.
I20251028 14:52:33.753479 6473 tablet_bootstrap.cc:654] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:33.756060 6473 tablet_bootstrap.cc:492] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f: No bootstrap required, opened a new log
I20251028 14:52:33.756273 6473 ts_tablet_manager.cc:1403] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20251028 14:52:33.757300 6473 raft_consensus.cc:359] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.757611 6473 raft_consensus.cc:385] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:33.757752 6473 raft_consensus.cc:740] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e55c6194c420086dc337d79f7cf5f, State: Initialized, Role: FOLLOWER
I20251028 14:52:33.758139 6473 consensus_queue.cc:260] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.758373 6473 raft_consensus.cc:399] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:33.758487 6473 raft_consensus.cc:493] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:33.758627 6473 raft_consensus.cc:3060] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:33.762035 6473 raft_consensus.cc:515] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.762506 6473 leader_election.cc:304] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6e55c6194c420086dc337d79f7cf5f; no voters:
I20251028 14:52:33.763144 6473 leader_election.cc:290] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:33.763337 6475 raft_consensus.cc:2804] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:33.764117 6473 ts_tablet_manager.cc:1434] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251028 14:52:33.764367 6463 heartbeater.cc:499] Master 127.0.104.62:41081 was elected leader, sending a full tablet report...
I20251028 14:52:33.764576 6473 tablet_bootstrap.cc:492] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f: Bootstrap starting.
I20251028 14:52:33.765574 6475 raft_consensus.cc:697] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 1 LEADER]: Becoming Leader. State: Replica: aa6e55c6194c420086dc337d79f7cf5f, State: Running, Role: LEADER
I20251028 14:52:33.765967 6475 consensus_queue.cc:237] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.767050 6473 tablet_bootstrap.cc:654] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:33.772765 6311 catalog_manager.cc:5649] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f reported cstate change: term changed from 0 to 1, leader changed from <none> to aa6e55c6194c420086dc337d79f7cf5f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aa6e55c6194c420086dc337d79f7cf5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:33.776010 6473 tablet_bootstrap.cc:492] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f: No bootstrap required, opened a new log
I20251028 14:52:33.776247 6473 ts_tablet_manager.cc:1403] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20251028 14:52:33.777184 6473 raft_consensus.cc:359] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.778383 6473 raft_consensus.cc:385] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:33.778489 6473 raft_consensus.cc:740] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e55c6194c420086dc337d79f7cf5f, State: Initialized, Role: FOLLOWER
I20251028 14:52:33.778816 6473 consensus_queue.cc:260] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.779011 6473 raft_consensus.cc:399] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:33.779105 6473 raft_consensus.cc:493] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:33.779212 6473 raft_consensus.cc:3060] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:33.782596 6473 raft_consensus.cc:515] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.782900 6473 leader_election.cc:304] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6e55c6194c420086dc337d79f7cf5f; no voters:
I20251028 14:52:33.783139 6473 leader_election.cc:290] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:33.783255 6477 raft_consensus.cc:2804] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:33.783459 6477 raft_consensus.cc:697] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 1 LEADER]: Becoming Leader. State: Replica: aa6e55c6194c420086dc337d79f7cf5f, State: Running, Role: LEADER
I20251028 14:52:33.783608 6473 ts_tablet_manager.cc:1434] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20251028 14:52:33.783830 6477 consensus_queue.cc:237] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:33.786973 6311 catalog_manager.cc:5649] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f reported cstate change: term changed from 0 to 1, leader changed from <none> to aa6e55c6194c420086dc337d79f7cf5f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aa6e55c6194c420086dc337d79f7cf5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:33.805445 6469 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:52:34.637238 6311 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46438:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:34.646136 6427 tablet_service.cc:1505] Processing CreateTablet for tablet cbeb6107a1c1492f9ca4dce42c0bd745 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c787150fa734d70a15abaeebdc49009]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:34.646615 6427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbeb6107a1c1492f9ca4dce42c0bd745. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:34.651805 6491 tablet_bootstrap.cc:492] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: Bootstrap starting.
I20251028 14:52:34.653403 6491 tablet_bootstrap.cc:654] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:34.655257 6491 tablet_bootstrap.cc:492] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: No bootstrap required, opened a new log
I20251028 14:52:34.655421 6491 ts_tablet_manager.cc:1403] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251028 14:52:34.656076 6491 raft_consensus.cc:359] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:34.656248 6491 raft_consensus.cc:385] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:34.656340 6491 raft_consensus.cc:740] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e55c6194c420086dc337d79f7cf5f, State: Initialized, Role: FOLLOWER
I20251028 14:52:34.656536 6491 consensus_queue.cc:260] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:34.656687 6491 raft_consensus.cc:399] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:34.656754 6491 raft_consensus.cc:493] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:34.656857 6491 raft_consensus.cc:3060] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:34.658830 6491 raft_consensus.cc:515] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:34.659096 6491 leader_election.cc:304] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa6e55c6194c420086dc337d79f7cf5f; no voters:
I20251028 14:52:34.659595 6491 leader_election.cc:290] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:34.659745 6493 raft_consensus.cc:2804] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:34.660477 6491 ts_tablet_manager.cc:1434] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20251028 14:52:34.660813 6493 raft_consensus.cc:697] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 1 LEADER]: Becoming Leader. State: Replica: aa6e55c6194c420086dc337d79f7cf5f, State: Running, Role: LEADER
I20251028 14:52:34.661108 6493 consensus_queue.cc:237] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } }
I20251028 14:52:34.662722 6495 tablet_replica.cc:442] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: TxnStatusTablet state changed. Reason: New leader aa6e55c6194c420086dc337d79f7cf5f. Latest consensus state: current_term: 1 leader_uuid: "aa6e55c6194c420086dc337d79f7cf5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } } }
I20251028 14:52:34.662679 6494 tablet_replica.cc:442] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa6e55c6194c420086dc337d79f7cf5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } } }
I20251028 14:52:34.663278 6494 tablet_replica.cc:445] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:34.664412 6311 catalog_manager.cc:5649] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f reported cstate change: term changed from 0 to 1, leader changed from <none> to aa6e55c6194c420086dc337d79f7cf5f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "aa6e55c6194c420086dc337d79f7cf5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa6e55c6194c420086dc337d79f7cf5f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45933 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:34.663228 6495 tablet_replica.cc:445] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:34.664886 6497 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:34.665197 6497 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251028 14:52:35.107285 6498 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251028 14:52:35.116365 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:52:35.130062 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:35.130998 416 tablet_replica.cc:333] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f: stopping tablet replica
I20251028 14:52:35.131524 416 raft_consensus.cc:2243] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:35.131883 416 raft_consensus.cc:2272] T 525ffdcfed074308943cbcc791fe6d85 P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:35.134019 416 tablet_replica.cc:333] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f: stopping tablet replica
I20251028 14:52:35.134177 416 raft_consensus.cc:2243] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:35.134415 416 raft_consensus.cc:2272] T cbeb6107a1c1492f9ca4dce42c0bd745 P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:35.135236 416 tablet_replica.cc:333] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f: stopping tablet replica
I20251028 14:52:35.135380 416 raft_consensus.cc:2243] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:35.135519 416 raft_consensus.cc:2272] T 1a1c45be1a874c8c8cb2b93ec3ab4b1d P aa6e55c6194c420086dc337d79f7cf5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:35.153443 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:52:35.161602 416 master.cc:561] Master@127.0.104.62:41081 shutting down...
I20251028 14:52:35.173110 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:35.173467 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:35.173645 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 52bbb805d93340e58f5b5309c0f82aed: stopping tablet replica
I20251028 14:52:35.188508 416 master.cc:583] Master@127.0.104.62:41081 shutdown complete.
[ OK ] TxnOpDispatcherITest.LifecycleBasic (1784 ms)
[ RUN ] TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable
I20251028 14:52:35.200984 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:41031
I20251028 14:52:35.201588 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:35.204043 6513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:35.204146 6514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:35.204522 6516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:35.204778 416 server_base.cc:1047] running on GCE node
I20251028 14:52:35.205056 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:35.205132 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:35.205188 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663155205170 us; error 0 us; skew 500 ppm
I20251028 14:52:35.205389 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:35.206678 416 webserver.cc:492] Webserver started at http://127.0.104.62:44301/ using document root <none> and password file <none>
I20251028 14:52:35.206974 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:35.207057 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:35.207214 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:35.207875 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "4cb8a02924b240059413238d911aa0b2"
format_stamp: "Formatted at 2025-10-28 14:52:35 on dist-test-slave-0v9m"
I20251028 14:52:35.210682 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20251028 14:52:35.212674 6521 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:35.213207 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20251028 14:52:35.213403 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "4cb8a02924b240059413238d911aa0b2"
format_stamp: "Formatted at 2025-10-28 14:52:35 on dist-test-slave-0v9m"
I20251028 14:52:35.213649 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:35.224198 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:35.224967 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:35.243161 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41031
I20251028 14:52:35.243299 6582 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41031 every 8 connection(s)
I20251028 14:52:35.244656 6583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:35.248955 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2: Bootstrap starting.
I20251028 14:52:35.250416 6583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:35.252213 6583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2: No bootstrap required, opened a new log
I20251028 14:52:35.252887 6583 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb8a02924b240059413238d911aa0b2" member_type: VOTER }
I20251028 14:52:35.253031 6583 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:35.253089 6583 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb8a02924b240059413238d911aa0b2, State: Initialized, Role: FOLLOWER
I20251028 14:52:35.253324 6583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb8a02924b240059413238d911aa0b2" member_type: VOTER }
I20251028 14:52:35.253468 6583 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:35.253528 6583 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:35.253613 6583 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:35.255515 6583 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb8a02924b240059413238d911aa0b2" member_type: VOTER }
I20251028 14:52:35.255748 6583 leader_election.cc:304] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cb8a02924b240059413238d911aa0b2; no voters:
I20251028 14:52:35.256189 6583 leader_election.cc:290] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:35.256356 6586 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:35.256843 6586 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4cb8a02924b240059413238d911aa0b2, State: Running, Role: LEADER
I20251028 14:52:35.257113 6583 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:35.257192 6586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb8a02924b240059413238d911aa0b2" member_type: VOTER }
I20251028 14:52:35.258189 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cb8a02924b240059413238d911aa0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb8a02924b240059413238d911aa0b2" member_type: VOTER } }
I20251028 14:52:35.258224 6588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4cb8a02924b240059413238d911aa0b2. Latest consensus state: current_term: 1 leader_uuid: "4cb8a02924b240059413238d911aa0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb8a02924b240059413238d911aa0b2" member_type: VOTER } }
I20251028 14:52:35.258785 6588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:35.258847 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:35.259865 6591 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:35.262476 6591 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:35.263077 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:52:35.267112 6591 catalog_manager.cc:1357] Generated new cluster ID: 9a0c371d0f3a49aa82bdca468eef06d3
I20251028 14:52:35.267230 6591 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:35.280676 6591 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:35.281915 6591 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:35.292984 6591 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2: Generated new TSK 0
I20251028 14:52:35.293424 6591 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:35.296304 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:35.299530 6610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:35.300172 6611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:35.301452 6613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:35.302009 416 server_base.cc:1047] running on GCE node
I20251028 14:52:35.301932 6538 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46332:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:35.302628 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:35.302721 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:35.302799 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663155302762 us; error 0 us; skew 500 ppm
I20251028 14:52:35.303035 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:35.304781 416 webserver.cc:492] Webserver started at http://127.0.104.1:37849/ using document root <none> and password file <none>
I20251028 14:52:35.305114 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:35.305226 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:35.305399 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:35.306093 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "ac5af5084ba44bd886f987a837581617"
format_stamp: "Formatted at 2025-10-28 14:52:35 on dist-test-slave-0v9m"
I20251028 14:52:35.308633 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251028 14:52:35.310652 6618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:35.311139 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:52:35.311270 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "ac5af5084ba44bd886f987a837581617"
format_stamp: "Formatted at 2025-10-28 14:52:35 on dist-test-slave-0v9m"
I20251028 14:52:35.311483 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:35.319396 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:35.320112 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:35.322058 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:35.322254 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:35.322379 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:35.322480 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:35.352865 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45505
I20251028 14:52:35.352874 6687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45505 every 8 connection(s)
I20251028 14:52:35.354779 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251028 14:52:35.355508 6688 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41031
I20251028 14:52:35.355672 6688 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:35.356171 6688 heartbeater.cc:507] Master 127.0.104.62:41031 requested a full tablet report, sending...
I20251028 14:52:35.357550 6538 ts_manager.cc:194] Registered new tserver with Master: ac5af5084ba44bd886f987a837581617 (127.0.104.1:45505)
W20251028 14:52:35.358197 6691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:35.358589 6690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:35.359431 6538 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46336
W20251028 14:52:35.361505 6693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:35.365583 416 server_base.cc:1047] running on GCE node
I20251028 14:52:35.365917 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:35.365993 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:35.366060 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663155366029 us; error 0 us; skew 500 ppm
I20251028 14:52:35.366218 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:35.367548 416 webserver.cc:492] Webserver started at http://127.0.104.2:42133/ using document root <none> and password file <none>
I20251028 14:52:35.367787 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:35.367887 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:35.368052 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:35.368711 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-1-root/instance:
uuid: "667bfbc38e0f4a68bc62c6585edb6f3a"
format_stamp: "Formatted at 2025-10-28 14:52:35 on dist-test-slave-0v9m"
I20251028 14:52:35.371937 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20251028 14:52:35.374636 6698 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:35.375162 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:35.375316 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-1-root
uuid: "667bfbc38e0f4a68bc62c6585edb6f3a"
format_stamp: "Formatted at 2025-10-28 14:52:35 on dist-test-slave-0v9m"
I20251028 14:52:35.375473 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:35.392853 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:35.393626 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:35.395686 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:35.395813 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:35.395960 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:35.396064 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:35.426688 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42927
I20251028 14:52:35.426779 6767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42927 every 8 connection(s)
I20251028 14:52:35.428812 6768 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41031
I20251028 14:52:35.428972 6768 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:35.429440 6768 heartbeater.cc:507] Master 127.0.104.62:41031 requested a full tablet report, sending...
I20251028 14:52:35.430349 6538 ts_manager.cc:194] Registered new tserver with Master: 667bfbc38e0f4a68bc62c6585edb6f3a (127.0.104.2:42927)
I20251028 14:52:35.430474 416 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.00266771s
I20251028 14:52:35.431293 6538 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46352
I20251028 14:52:36.319666 6538 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39000:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:36.338645 6653 tablet_service.cc:1505] Processing CreateTablet for tablet 3b5d549c85234239b61d785bb71bc676 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8cf370539e8a4ac78de473a745a49695]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:36.339227 6653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b5d549c85234239b61d785bb71bc676. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:36.344904 6788 tablet_bootstrap.cc:492] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Bootstrap starting.
I20251028 14:52:36.346935 6788 tablet_bootstrap.cc:654] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:36.349090 6788 tablet_bootstrap.cc:492] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: No bootstrap required, opened a new log
I20251028 14:52:36.349308 6788 ts_tablet_manager.cc:1403] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:52:36.349992 6788 raft_consensus.cc:359] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:36.350142 6788 raft_consensus.cc:385] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:36.350226 6788 raft_consensus.cc:740] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5af5084ba44bd886f987a837581617, State: Initialized, Role: FOLLOWER
I20251028 14:52:36.350435 6788 consensus_queue.cc:260] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:36.350595 6788 raft_consensus.cc:399] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:36.350672 6788 raft_consensus.cc:493] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:36.350885 6788 raft_consensus.cc:3060] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:36.353912 6788 raft_consensus.cc:515] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:36.354234 6788 leader_election.cc:304] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5af5084ba44bd886f987a837581617; no voters:
I20251028 14:52:36.354779 6788 leader_election.cc:290] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:36.354995 6790 raft_consensus.cc:2804] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:36.355563 6788 ts_tablet_manager.cc:1434] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251028 14:52:36.355680 6688 heartbeater.cc:499] Master 127.0.104.62:41031 was elected leader, sending a full tablet report...
I20251028 14:52:36.356062 6790 raft_consensus.cc:697] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 LEADER]: Becoming Leader. State: Replica: ac5af5084ba44bd886f987a837581617, State: Running, Role: LEADER
I20251028 14:52:36.356339 6790 consensus_queue.cc:237] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:36.357340 6791 tablet_replica.cc:442] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac5af5084ba44bd886f987a837581617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } } }
I20251028 14:52:36.357609 6791 tablet_replica.cc:445] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:36.357740 6792 tablet_replica.cc:442] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: TxnStatusTablet state changed. Reason: New leader ac5af5084ba44bd886f987a837581617. Latest consensus state: current_term: 1 leader_uuid: "ac5af5084ba44bd886f987a837581617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } } }
I20251028 14:52:36.358697 6792 tablet_replica.cc:445] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:36.359593 6794 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:36.359786 6794 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:36.360190 6538 catalog_manager.cc:5649] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac5af5084ba44bd886f987a837581617 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ac5af5084ba44bd886f987a837581617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:36.432848 6768 heartbeater.cc:499] Master 127.0.104.62:41031 was elected leader, sending a full tablet report...
I20251028 14:52:36.471050 6538 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46370:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251028 14:52:36.484858 6733 tablet_service.cc:1505] Processing CreateTablet for tablet dd18676c706e4a8a856272558d7f1e63 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=cb074b36079445c1b84897b6aceed56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:36.485378 6733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd18676c706e4a8a856272558d7f1e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:36.490965 6797 tablet_bootstrap.cc:492] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a: Bootstrap starting.
I20251028 14:52:36.492620 6797 tablet_bootstrap.cc:654] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:36.495187 6797 tablet_bootstrap.cc:492] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a: No bootstrap required, opened a new log
I20251028 14:52:36.495375 6797 ts_tablet_manager.cc:1403] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251028 14:52:36.496102 6797 raft_consensus.cc:359] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667bfbc38e0f4a68bc62c6585edb6f3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42927 } }
I20251028 14:52:36.496258 6797 raft_consensus.cc:385] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:36.496342 6797 raft_consensus.cc:740] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667bfbc38e0f4a68bc62c6585edb6f3a, State: Initialized, Role: FOLLOWER
I20251028 14:52:36.496563 6797 consensus_queue.cc:260] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667bfbc38e0f4a68bc62c6585edb6f3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42927 } }
I20251028 14:52:36.496956 6797 raft_consensus.cc:399] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:36.497080 6797 raft_consensus.cc:493] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:36.497383 6797 raft_consensus.cc:3060] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:36.499866 6797 raft_consensus.cc:515] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667bfbc38e0f4a68bc62c6585edb6f3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42927 } }
I20251028 14:52:36.500138 6797 leader_election.cc:304] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 667bfbc38e0f4a68bc62c6585edb6f3a; no voters:
I20251028 14:52:36.500640 6797 leader_election.cc:290] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:36.500864 6799 raft_consensus.cc:2804] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:36.501431 6797 ts_tablet_manager.cc:1434] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20251028 14:52:36.501605 6799 raft_consensus.cc:697] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 1 LEADER]: Becoming Leader. State: Replica: 667bfbc38e0f4a68bc62c6585edb6f3a, State: Running, Role: LEADER
I20251028 14:52:36.501889 6799 consensus_queue.cc:237] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667bfbc38e0f4a68bc62c6585edb6f3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42927 } }
I20251028 14:52:36.504297 6538 catalog_manager.cc:5649] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 667bfbc38e0f4a68bc62c6585edb6f3a (127.0.104.2). New cstate: current_term: 1 leader_uuid: "667bfbc38e0f4a68bc62c6585edb6f3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "667bfbc38e0f4a68bc62c6585edb6f3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42927 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:37.568287 6803 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20251028 14:52:37.569007 6803 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39102: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:52:37.569101 6812 ts_tablet_manager.cc:2121] Sending abort request for transaction 1
I20251028 14:52:37.569536 6803 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39102 (ReqId={client: 9a9166a4a5af42638ea7f07c151987e1, seq_no=1, attempt_no=0}) took 1008 ms. Trace:
I20251028 14:52:37.569762 6803 rpcz_store.cc:276] 1028 14:52:36.561504 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:36.561611 (+ 107us) service_pool.cc:225] Handling call
1028 14:52:37.569470 (+1007859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:36.566034 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:36.566105 (+ 71us) participant_op.cc:199] PREPARE: finished
1028 14:52:36.566131 (+ 26us) participant_op.cc:212] START. Timestamp: 1761663156566118
1028 14:52:36.566133 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:52:36.566255 (+ 122us) log.cc:844] Serialized 65 byte log entry
1028 14:52:36.566725 (+ 470us) op_driver.cc:464] REPLICATION: finished
1028 14:52:36.566842 (+ 117us) participant_op.cc:265] APPLY: starting
1028 14:52:36.567071 (+ 229us) participant_op.cc:270] APPLY: finished
1028 14:52:36.567102 (+ 31us) log.cc:844] Serialized 14 byte log entry
1028 14:52:36.567142 (+ 40us) participant_op.cc:94] Released txn lock
1028 14:52:36.567149 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:52:36.567450 (+ 0us) write_op.cc:183] PREPARE: starting on tablet dd18676c706e4a8a856272558d7f1e63
1028 14:52:36.567551 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:36.567554 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:36.567555 (+ 1us) tablet.cc:660] Decoding operations
1028 14:52:36.567635 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:37.568485 (+1000850us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:52:37.568499 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:37.568507 (+ 8us) write_op.cc:454] Released schema lock
1028 14:52:37.568514 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":72,"txn-participant-registration.run_cpu_time_us":619,"txn-participant-registration.run_wall_time_us":3837,"child_traces":[["op",{"apply.queue_time_us":59,"apply.run_cpu_time_us":576,"apply.run_wall_time_us":576,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":273,"prepare.run_wall_time_us":274,"replication_time_us":570,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":58,"thread_start_us":258,"threads_started":1,"txn-participant-registration.queue_time_us":383}]]}]]}
W20251028 14:52:37.572364 6771 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet dd18676c706e4a8a856272558d7f1e63 after 1 attempt(s): Failed to write to server: 667bfbc38e0f4a68bc62c6585edb6f3a (127.0.104.2:42927): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:52:37.572755 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:52:37.582499 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:37.582906 416 tablet_replica.cc:333] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: stopping tablet replica
I20251028 14:52:37.583169 416 raft_consensus.cc:2243] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:37.583428 416 raft_consensus.cc:2272] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251028 14:52:37.589187 6803 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations [suppressed 1 similar messages]
I20251028 14:52:37.634874 416 txn_status_manager.cc:765] Waiting for 1 task(s) to stop
I20251028 14:52:37.649163 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:52:40.654345 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:40.657140 6823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:40.657306 6824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:40.658110 6826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:40.658886 416 server_base.cc:1047] running on GCE node
I20251028 14:52:40.659339 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:40.659427 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:40.659516 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663160659477 us; error 0 us; skew 500 ppm
I20251028 14:52:40.659680 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:40.661357 416 webserver.cc:492] Webserver started at http://127.0.104.1:37849/ using document root <none> and password file <none>
I20251028 14:52:40.661608 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:40.661686 416 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251028 14:52:40.663702 416 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251028 14:52:40.665584 6831 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:40.665948 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251028 14:52:40.666054 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "ac5af5084ba44bd886f987a837581617"
format_stamp: "Formatted at 2025-10-28 14:52:35 on dist-test-slave-0v9m"
I20251028 14:52:40.666186 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:40.681063 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:40.681725 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:40.684340 6840 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20251028 14:52:40.688241 416 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251028 14:52:40.688410 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20251028 14:52:40.688545 416 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251028 14:52:40.690606 416 ts_tablet_manager.cc:616] Registered 1 tablets
I20251028 14:52:40.690686 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20251028 14:52:40.690798 6840 tablet_bootstrap.cc:492] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Bootstrap starting.
I20251028 14:52:40.704466 6840 tablet_bootstrap.cc:492] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=4 ignored=0} mutations{seen=1 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251028 14:52:40.705839 6840 tablet_bootstrap.cc:492] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Bootstrap complete.
I20251028 14:52:40.706251 6840 ts_tablet_manager.cc:1403] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.004s
I20251028 14:52:40.707199 6840 raft_consensus.cc:359] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:40.707469 6840 raft_consensus.cc:740] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac5af5084ba44bd886f987a837581617, State: Initialized, Role: FOLLOWER
I20251028 14:52:40.707719 6840 consensus_queue.cc:260] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:40.707878 6840 raft_consensus.cc:399] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:40.707975 6840 raft_consensus.cc:493] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:40.708091 6840 raft_consensus.cc:3060] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 1 FOLLOWER]: Advancing to term 2
I20251028 14:52:40.710527 6840 raft_consensus.cc:515] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:40.710809 6840 leader_election.cc:304] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac5af5084ba44bd886f987a837581617; no voters:
I20251028 14:52:40.711375 6840 leader_election.cc:290] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251028 14:52:40.711499 6878 raft_consensus.cc:2804] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 2 FOLLOWER]: Leader election won for term 2
I20251028 14:52:40.715052 6840 ts_tablet_manager.cc:1434] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617: Time spent starting tablet: real 0.009s user 0.001s sys 0.007s
I20251028 14:52:40.715142 6878 raft_consensus.cc:697] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 2 LEADER]: Becoming Leader. State: Replica: ac5af5084ba44bd886f987a837581617, State: Running, Role: LEADER
I20251028 14:52:40.715629 6878 consensus_queue.cc:237] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } }
I20251028 14:52:40.716787 6879 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ac5af5084ba44bd886f987a837581617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } } }
I20251028 14:52:40.717022 6879 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:40.718101 6880 tablet_replica.cc:442] TxnStatusTablet state changed. Reason: New leader ac5af5084ba44bd886f987a837581617. Latest consensus state: current_term: 2 leader_uuid: "ac5af5084ba44bd886f987a837581617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } } }
I20251028 14:52:40.718393 6880 tablet_replica.cc:445] This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:40.718278 6886 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:40.718734 6886 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:40.721937 6886 txn_status_manager.cc:728] Starting 1 aborts task
I20251028 14:52:40.729507 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45505
I20251028 14:52:40.729564 6911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45505 every 8 connection(s)
I20251028 14:52:40.731741 6912 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41031
I20251028 14:52:40.731920 6912 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:40.732456 6912 heartbeater.cc:507] Master 127.0.104.62:41031 requested a full tablet report, sending...
W20251028 14:52:40.734519 6916 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:39120: Illegal state: Transaction metadata for transaction 1 already set
I20251028 14:52:40.734570 6538 ts_manager.cc:194] Re-registered known tserver with Master: ac5af5084ba44bd886f987a837581617 (127.0.104.1:45505)
I20251028 14:52:40.735216 6538 catalog_manager.cc:5649] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "ac5af5084ba44bd886f987a837581617" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac5af5084ba44bd886f987a837581617" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45505 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:40.744029 416 tablet_server.cc:178] TabletServer@127.0.104.1:45505 shutting down...
I20251028 14:52:40.747447 6538 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39008
I20251028 14:52:40.750380 6912 heartbeater.cc:499] Master 127.0.104.62:41031 was elected leader, sending a full tablet report...
I20251028 14:52:40.756134 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:40.756556 416 tablet_replica.cc:333] stopping tablet replica
I20251028 14:52:40.756778 416 raft_consensus.cc:2243] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 2 LEADER]: Raft consensus shutting down.
I20251028 14:52:40.757017 416 raft_consensus.cc:2272] T 3b5d549c85234239b61d785bb71bc676 P ac5af5084ba44bd886f987a837581617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:40.773284 416 tablet_server.cc:195] TabletServer@127.0.104.1:45505 shutdown complete.
I20251028 14:52:40.778254 416 tablet_server.cc:178] TabletServer@127.0.104.2:0 shutting down...
I20251028 14:52:40.786814 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:40.787106 416 tablet_replica.cc:333] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a: stopping tablet replica
I20251028 14:52:40.787274 416 raft_consensus.cc:2243] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:40.787446 416 raft_consensus.cc:2272] T dd18676c706e4a8a856272558d7f1e63 P 667bfbc38e0f4a68bc62c6585edb6f3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:40.801685 416 tablet_server.cc:195] TabletServer@127.0.104.2:0 shutdown complete.
I20251028 14:52:40.807281 416 master.cc:561] Master@127.0.104.62:41031 shutting down...
I20251028 14:52:40.816826 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:40.817109 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:40.817268 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4cb8a02924b240059413238d911aa0b2: stopping tablet replica
I20251028 14:52:40.831838 416 master.cc:583] Master@127.0.104.62:41031 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable (5641 ms)
[ RUN ] TxnOpDispatcherITest.BeginTxnLockAbort
I20251028 14:52:40.842880 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:37035
I20251028 14:52:40.843418 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:40.845896 6923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:40.845984 6924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:40.846941 6926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:40.846930 416 server_base.cc:1047] running on GCE node
I20251028 14:52:40.847401 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:40.847484 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:40.847519 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663160847506 us; error 0 us; skew 500 ppm
I20251028 14:52:40.847658 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:40.849076 416 webserver.cc:492] Webserver started at http://127.0.104.62:46779/ using document root <none> and password file <none>
I20251028 14:52:40.849378 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:40.849457 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:40.849587 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:40.850289 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "7df7c27e1fc04be68eedd821f375333a"
format_stamp: "Formatted at 2025-10-28 14:52:40 on dist-test-slave-0v9m"
I20251028 14:52:40.852991 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20251028 14:52:40.855418 6931 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:40.855921 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251028 14:52:40.856096 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "7df7c27e1fc04be68eedd821f375333a"
format_stamp: "Formatted at 2025-10-28 14:52:40 on dist-test-slave-0v9m"
I20251028 14:52:40.856248 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:40.870307 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:40.870962 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:40.888450 6992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37035 every 8 connection(s)
I20251028 14:52:40.888499 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37035
I20251028 14:52:40.890053 6993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:40.894258 6993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a: Bootstrap starting.
I20251028 14:52:40.895697 6993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:40.897596 6993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a: No bootstrap required, opened a new log
I20251028 14:52:40.898269 6993 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df7c27e1fc04be68eedd821f375333a" member_type: VOTER }
I20251028 14:52:40.898408 6993 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:40.898466 6993 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7df7c27e1fc04be68eedd821f375333a, State: Initialized, Role: FOLLOWER
I20251028 14:52:40.898660 6993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df7c27e1fc04be68eedd821f375333a" member_type: VOTER }
I20251028 14:52:40.898794 6993 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:40.898844 6993 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:40.898918 6993 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:40.900792 6993 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df7c27e1fc04be68eedd821f375333a" member_type: VOTER }
I20251028 14:52:40.901005 6993 leader_election.cc:304] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7df7c27e1fc04be68eedd821f375333a; no voters:
I20251028 14:52:40.901630 6993 leader_election.cc:290] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:40.901933 6996 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:40.902588 6993 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:40.902595 6996 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 1 LEADER]: Becoming Leader. State: Replica: 7df7c27e1fc04be68eedd821f375333a, State: Running, Role: LEADER
I20251028 14:52:40.902959 6996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df7c27e1fc04be68eedd821f375333a" member_type: VOTER }
I20251028 14:52:40.904067 6997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7df7c27e1fc04be68eedd821f375333a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df7c27e1fc04be68eedd821f375333a" member_type: VOTER } }
I20251028 14:52:40.904093 6998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7df7c27e1fc04be68eedd821f375333a. Latest consensus state: current_term: 1 leader_uuid: "7df7c27e1fc04be68eedd821f375333a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7df7c27e1fc04be68eedd821f375333a" member_type: VOTER } }
I20251028 14:52:40.904327 6997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:40.904330 6998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:40.904963 7004 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:40.907119 7004 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:40.907678 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:52:40.911149 7004 catalog_manager.cc:1357] Generated new cluster ID: 777ae5522e6445b79e78f0604bcf4d0f
I20251028 14:52:40.911270 7004 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:40.940425 7004 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:40.941229 7004 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:40.950562 7004 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a: Generated new TSK 0
I20251028 14:52:40.950822 7004 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:40.959683 6948 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34454:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:40.972965 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:40.976044 7021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:40.976086 7020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:40.976893 7023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:40.976943 416 server_base.cc:1047] running on GCE node
I20251028 14:52:40.977332 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:40.977406 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:40.977458 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663160977434 us; error 0 us; skew 500 ppm
I20251028 14:52:40.977588 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:40.978847 416 webserver.cc:492] Webserver started at http://127.0.104.1:41185/ using document root <none> and password file <none>
I20251028 14:52:40.979069 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:40.979151 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:40.979261 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:40.979866 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "506ea171d559433dac60a76935710cff"
format_stamp: "Formatted at 2025-10-28 14:52:40 on dist-test-slave-0v9m"
I20251028 14:52:40.982476 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251028 14:52:40.984231 7028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:40.984630 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251028 14:52:40.984736 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "506ea171d559433dac60a76935710cff"
format_stamp: "Formatted at 2025-10-28 14:52:40 on dist-test-slave-0v9m"
I20251028 14:52:40.984874 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:41.011970 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:41.012684 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:41.014588 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:41.014698 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:41.014868 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:41.014969 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:41.041671 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34899
I20251028 14:52:41.041741 7097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34899 every 8 connection(s)
I20251028 14:52:41.043818 7098 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37035
I20251028 14:52:41.043988 7098 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:41.044421 7098 heartbeater.cc:507] Master 127.0.104.62:37035 requested a full tablet report, sending...
I20251028 14:52:41.045459 6948 ts_manager.cc:194] Registered new tserver with Master: 506ea171d559433dac60a76935710cff (127.0.104.1:34899)
I20251028 14:52:41.046133 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003569365s
I20251028 14:52:41.046900 6948 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34456
I20251028 14:52:41.060178 6948 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34468:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:52:41.076699 7062 tablet_service.cc:1505] Processing CreateTablet for tablet 4aaaea279bd54c8fa95779fde590dbdc (DEFAULT_TABLE table=txn_op_dispatcher_test [id=220bf6ae6ff74975b24f6da4077e0bbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:41.076704 7061 tablet_service.cc:1505] Processing CreateTablet for tablet 6fc9acb7963b4702a8b0fbec1f39448e (DEFAULT_TABLE table=txn_op_dispatcher_test [id=220bf6ae6ff74975b24f6da4077e0bbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:41.077267 7062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aaaea279bd54c8fa95779fde590dbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:41.077512 7061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fc9acb7963b4702a8b0fbec1f39448e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:41.083477 7108 tablet_bootstrap.cc:492] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff: Bootstrap starting.
I20251028 14:52:41.085443 7108 tablet_bootstrap.cc:654] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:41.087435 7108 tablet_bootstrap.cc:492] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff: No bootstrap required, opened a new log
I20251028 14:52:41.087599 7108 ts_tablet_manager.cc:1403] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251028 14:52:41.089017 7108 raft_consensus.cc:359] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.089291 7108 raft_consensus.cc:385] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:41.089607 7108 raft_consensus.cc:740] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506ea171d559433dac60a76935710cff, State: Initialized, Role: FOLLOWER
I20251028 14:52:41.090358 7108 consensus_queue.cc:260] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.090745 7108 raft_consensus.cc:399] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:41.090839 7108 raft_consensus.cc:493] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:41.090974 7108 raft_consensus.cc:3060] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:41.093200 7108 raft_consensus.cc:515] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.093559 7108 leader_election.cc:304] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506ea171d559433dac60a76935710cff; no voters:
I20251028 14:52:41.094029 7108 leader_election.cc:290] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:41.094252 7110 raft_consensus.cc:2804] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:41.094765 7108 ts_tablet_manager.cc:1434] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20251028 14:52:41.094775 7098 heartbeater.cc:499] Master 127.0.104.62:37035 was elected leader, sending a full tablet report...
I20251028 14:52:41.094871 7110 raft_consensus.cc:697] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 1 LEADER]: Becoming Leader. State: Replica: 506ea171d559433dac60a76935710cff, State: Running, Role: LEADER
I20251028 14:52:41.095201 7108 tablet_bootstrap.cc:492] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff: Bootstrap starting.
I20251028 14:52:41.095273 7110 consensus_queue.cc:237] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.097764 7108 tablet_bootstrap.cc:654] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:41.097867 6947 catalog_manager.cc:5649] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff reported cstate change: term changed from 0 to 1, leader changed from <none> to 506ea171d559433dac60a76935710cff (127.0.104.1). New cstate: current_term: 1 leader_uuid: "506ea171d559433dac60a76935710cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:41.099653 7108 tablet_bootstrap.cc:492] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff: No bootstrap required, opened a new log
I20251028 14:52:41.099861 7108 ts_tablet_manager.cc:1403] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20251028 14:52:41.100569 7108 raft_consensus.cc:359] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.100742 7108 raft_consensus.cc:385] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:41.101420 7108 raft_consensus.cc:740] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506ea171d559433dac60a76935710cff, State: Initialized, Role: FOLLOWER
I20251028 14:52:41.101702 7108 consensus_queue.cc:260] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.101878 7108 raft_consensus.cc:399] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:41.101989 7108 raft_consensus.cc:493] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:41.102097 7108 raft_consensus.cc:3060] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:41.104630 7108 raft_consensus.cc:515] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.105026 7108 leader_election.cc:304] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506ea171d559433dac60a76935710cff; no voters:
I20251028 14:52:41.105829 7110 raft_consensus.cc:2804] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:41.105829 7108 leader_election.cc:290] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:41.106117 7110 raft_consensus.cc:697] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 1 LEADER]: Becoming Leader. State: Replica: 506ea171d559433dac60a76935710cff, State: Running, Role: LEADER
I20251028 14:52:41.106531 7108 ts_tablet_manager.cc:1434] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251028 14:52:41.106498 7110 consensus_queue.cc:237] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.109982 6948 catalog_manager.cc:5649] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff reported cstate change: term changed from 0 to 1, leader changed from <none> to 506ea171d559433dac60a76935710cff (127.0.104.1). New cstate: current_term: 1 leader_uuid: "506ea171d559433dac60a76935710cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:41.118795 7104 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:52:41.977519 6947 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34480:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:41.986318 7061 tablet_service.cc:1505] Processing CreateTablet for tablet 74256d63aa984cc4a566d6bcc57bb0e2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=11f7473509844cd58a506f1ef5e34a59]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:41.986961 7061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74256d63aa984cc4a566d6bcc57bb0e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:41.992033 7126 tablet_bootstrap.cc:492] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: Bootstrap starting.
I20251028 14:52:41.993849 7126 tablet_bootstrap.cc:654] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:41.995935 7126 tablet_bootstrap.cc:492] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: No bootstrap required, opened a new log
I20251028 14:52:41.996106 7126 ts_tablet_manager.cc:1403] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:41.996809 7126 raft_consensus.cc:359] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.996989 7126 raft_consensus.cc:385] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:41.997140 7126 raft_consensus.cc:740] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506ea171d559433dac60a76935710cff, State: Initialized, Role: FOLLOWER
I20251028 14:52:41.997390 7126 consensus_queue.cc:260] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.997529 7126 raft_consensus.cc:399] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:41.997586 7126 raft_consensus.cc:493] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:41.997656 7126 raft_consensus.cc:3060] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:41.999531 7126 raft_consensus.cc:515] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:41.999748 7126 leader_election.cc:304] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 506ea171d559433dac60a76935710cff; no voters:
I20251028 14:52:42.000193 7126 leader_election.cc:290] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:42.000406 7128 raft_consensus.cc:2804] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:42.001322 7126 ts_tablet_manager.cc:1434] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251028 14:52:42.001585 7128 raft_consensus.cc:697] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 1 LEADER]: Becoming Leader. State: Replica: 506ea171d559433dac60a76935710cff, State: Running, Role: LEADER
I20251028 14:52:42.001899 7128 consensus_queue.cc:237] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } }
I20251028 14:52:42.003383 7129 tablet_replica.cc:442] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "506ea171d559433dac60a76935710cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } } }
I20251028 14:52:42.003505 7130 tablet_replica.cc:442] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: TxnStatusTablet state changed. Reason: New leader 506ea171d559433dac60a76935710cff. Latest consensus state: current_term: 1 leader_uuid: "506ea171d559433dac60a76935710cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } } }
I20251028 14:52:42.003779 7129 tablet_replica.cc:445] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:42.003787 7130 tablet_replica.cc:445] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:42.005638 6947 catalog_manager.cc:5649] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff reported cstate change: term changed from 0 to 1, leader changed from <none> to 506ea171d559433dac60a76935710cff (127.0.104.1). New cstate: current_term: 1 leader_uuid: "506ea171d559433dac60a76935710cff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "506ea171d559433dac60a76935710cff" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34899 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:42.006234 7132 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:42.006456 7132 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251028 14:52:43.676519 7137 lock_manager.cc:494] Txn 9223372036854775807 has not acquired the partition lock after 1000ms
W20251028 14:52:43.676810 7137 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46434: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:52:43.677155 7137 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46434 (ReqId={client: 3b57ba0958af4a9093a5291b7ad48706, seq_no=6, attempt_no=0}) took 1002 ms. Trace:
I20251028 14:52:43.677232 7137 rpcz_store.cc:276] 1028 14:52:42.675069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:42.675172 (+ 103us) service_pool.cc:225] Handling call
1028 14:52:43.677088 (+1001916us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:42.675692 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6fc9acb7963b4702a8b0fbec1f39448e
1028 14:52:42.675787 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:42.675791 (+ 4us) write_op.cc:435] Acquired schema lock
1028 14:52:42.675793 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:42.675864 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:43.676734 (+1000870us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:52:43.676760 (+ 26us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:43.676768 (+ 8us) write_op.cc:454] Released schema lock
1028 14:52:43.676778 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":63}]]}
W20251028 14:52:43.677963 7102 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 6fc9acb7963b4702a8b0fbec1f39448e after 1 attempt(s): Failed to write to server: 506ea171d559433dac60a76935710cff (127.0.104.1:34899): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:52:44.692764 7137 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20251028 14:52:44.693517 7137 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46434: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20251028 14:52:44.693609 7152 ts_tablet_manager.cc:2121] Sending abort request for transaction 1
I20251028 14:52:44.693832 7137 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:46434 (ReqId={client: 3b57ba0958af4a9093a5291b7ad48706, seq_no=7, attempt_no=0}) took 1008 ms. Trace:
I20251028 14:52:44.693917 7137 rpcz_store.cc:276] 1028 14:52:43.685013 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:43.685103 (+ 90us) service_pool.cc:225] Handling call
1028 14:52:44.693776 (+1008673us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:43.690083 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:43.690120 (+ 37us) participant_op.cc:199] PREPARE: finished
1028 14:52:43.690149 (+ 29us) participant_op.cc:212] START. Timestamp: 1761663163690135
1028 14:52:43.690158 (+ 9us) op_driver.cc:348] REPLICATION: starting
1028 14:52:43.690272 (+ 114us) log.cc:844] Serialized 65 byte log entry
1028 14:52:43.691127 (+ 855us) op_driver.cc:464] REPLICATION: finished
1028 14:52:43.691280 (+ 153us) participant_op.cc:265] APPLY: starting
1028 14:52:43.691600 (+ 320us) participant_op.cc:270] APPLY: finished
1028 14:52:43.691636 (+ 36us) log.cc:844] Serialized 14 byte log entry
1028 14:52:43.691672 (+ 36us) participant_op.cc:94] Released txn lock
1028 14:52:43.691679 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:52:43.691962 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 4aaaea279bd54c8fa95779fde590dbdc
1028 14:52:43.692076 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:43.692078 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:52:43.692080 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:43.692155 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:44.692935 (+1000780us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
1028 14:52:44.692950 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:44.692957 (+ 7us) write_op.cc:454] Released schema lock
1028 14:52:44.692964 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":219,"threads_started":1,"txn-participant-registration.queue_time_us":329,"txn-participant-registration.run_cpu_time_us":624,"txn-participant-registration.run_wall_time_us":4273,"child_traces":[["op",{"apply.queue_time_us":77,"apply.run_cpu_time_us":663,"apply.run_wall_time_us":663,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":596,"prepare.run_wall_time_us":595,"raft.queue_time_us":62,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":103,"replication_time_us":948,"thread_start_us":262,"threads_started":1,"wal-append.queue_time_us":359,"wal-append.run_cpu_time_us":865,"wal-append.run_wall_time_us":1001545,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":57,"thread_start_us":447,"threads_started":1,"txn-participant-registration.queue_time_us":547}]]}]]}
W20251028 14:52:44.694911 7102 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 4aaaea279bd54c8fa95779fde590dbdc after 1 attempt(s): Failed to write to server: 506ea171d559433dac60a76935710cff (127.0.104.1:34899): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20251028 14:52:44.700279 7137 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
W20251028 14:52:44.725129 7152 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:46434: Illegal state: Failed to write to server: 506ea171d559433dac60a76935710cff (127.0.104.1:34899): transaction ID 1 not open: ABORTED
I20251028 14:52:44.729276 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:52:44.739192 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:44.739687 416 tablet_replica.cc:333] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff: stopping tablet replica
I20251028 14:52:44.739908 416 raft_consensus.cc:2243] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:44.740206 416 raft_consensus.cc:2272] T 74256d63aa984cc4a566d6bcc57bb0e2 P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:44.741009 416 tablet_replica.cc:333] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff: stopping tablet replica
I20251028 14:52:44.741180 416 raft_consensus.cc:2243] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:44.741413 416 raft_consensus.cc:2272] T 6fc9acb7963b4702a8b0fbec1f39448e P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:44.742156 416 tablet_replica.cc:333] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff: stopping tablet replica
I20251028 14:52:44.742300 416 raft_consensus.cc:2243] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:44.742444 416 raft_consensus.cc:2272] T 4aaaea279bd54c8fa95779fde590dbdc P 506ea171d559433dac60a76935710cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:44.759281 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:52:44.764443 416 master.cc:561] Master@127.0.104.62:37035 shutting down...
I20251028 14:52:44.774514 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:44.774786 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:44.774904 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7df7c27e1fc04be68eedd821f375333a: stopping tablet replica
I20251028 14:52:44.788573 416 master.cc:583] Master@127.0.104.62:37035 shutdown complete.
[ OK ] TxnOpDispatcherITest.BeginTxnLockAbort (3958 ms)
[ RUN ] TxnOpDispatcherITest.BeginTxnLockRetry
I20251028 14:52:44.801982 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:33005
I20251028 14:52:44.802636 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:44.805456 7162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:44.805552 7163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:44.806394 7165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:44.806963 416 server_base.cc:1047] running on GCE node
I20251028 14:52:44.807430 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:44.807528 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:44.807576 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663164807554 us; error 0 us; skew 500 ppm
I20251028 14:52:44.807787 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:44.809559 416 webserver.cc:492] Webserver started at http://127.0.104.62:42725/ using document root <none> and password file <none>
I20251028 14:52:44.809836 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:44.809918 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:44.810071 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:44.810802 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "c47480ef006f47ce8d8192f3561f71ae"
format_stamp: "Formatted at 2025-10-28 14:52:44 on dist-test-slave-0v9m"
I20251028 14:52:44.813616 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20251028 14:52:44.815572 7170 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:44.816049 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251028 14:52:44.816207 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "c47480ef006f47ce8d8192f3561f71ae"
format_stamp: "Formatted at 2025-10-28 14:52:44 on dist-test-slave-0v9m"
I20251028 14:52:44.816344 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:44.856673 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:44.857370 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:44.873584 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33005
I20251028 14:52:44.873607 7231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33005 every 8 connection(s)
I20251028 14:52:44.874994 7232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:44.879317 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae: Bootstrap starting.
I20251028 14:52:44.880717 7232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:44.882472 7232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae: No bootstrap required, opened a new log
I20251028 14:52:44.883143 7232 raft_consensus.cc:359] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47480ef006f47ce8d8192f3561f71ae" member_type: VOTER }
I20251028 14:52:44.883282 7232 raft_consensus.cc:385] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:44.883356 7232 raft_consensus.cc:740] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47480ef006f47ce8d8192f3561f71ae, State: Initialized, Role: FOLLOWER
I20251028 14:52:44.883556 7232 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47480ef006f47ce8d8192f3561f71ae" member_type: VOTER }
I20251028 14:52:44.883695 7232 raft_consensus.cc:399] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:44.883750 7232 raft_consensus.cc:493] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:44.883827 7232 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:44.885700 7232 raft_consensus.cc:515] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47480ef006f47ce8d8192f3561f71ae" member_type: VOTER }
I20251028 14:52:44.885911 7232 leader_election.cc:304] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c47480ef006f47ce8d8192f3561f71ae; no voters:
I20251028 14:52:44.886279 7232 leader_election.cc:290] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:44.886482 7235 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:44.886924 7235 raft_consensus.cc:697] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 1 LEADER]: Becoming Leader. State: Replica: c47480ef006f47ce8d8192f3561f71ae, State: Running, Role: LEADER
I20251028 14:52:44.887212 7232 sys_catalog.cc:565] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:44.887188 7235 consensus_queue.cc:237] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47480ef006f47ce8d8192f3561f71ae" member_type: VOTER }
I20251028 14:52:44.888378 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c47480ef006f47ce8d8192f3561f71ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47480ef006f47ce8d8192f3561f71ae" member_type: VOTER } }
I20251028 14:52:44.888430 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader c47480ef006f47ce8d8192f3561f71ae. Latest consensus state: current_term: 1 leader_uuid: "c47480ef006f47ce8d8192f3561f71ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47480ef006f47ce8d8192f3561f71ae" member_type: VOTER } }
I20251028 14:52:44.888680 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:44.888660 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:44.890117 7240 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:44.891557 7240 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:44.892261 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:52:44.895231 7240 catalog_manager.cc:1357] Generated new cluster ID: 3df60d6bec6e4372b865223e7a17716b
I20251028 14:52:44.895370 7240 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:44.922421 7240 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:44.923422 7240 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:44.938714 7240 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae: Generated new TSK 0
I20251028 14:52:44.938987 7240 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:44.949992 7187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40842:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:44.958146 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:44.961131 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:44.961232 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:44.962147 416 server_base.cc:1047] running on GCE node
W20251028 14:52:44.962221 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:44.962606 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:44.962675 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:44.962746 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663164962713 us; error 0 us; skew 500 ppm
I20251028 14:52:44.963008 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:44.964407 416 webserver.cc:492] Webserver started at http://127.0.104.1:39985/ using document root <none> and password file <none>
I20251028 14:52:44.964655 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:44.964744 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:44.964887 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:44.965606 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "4cb3d6dfd43e4562bfd8ebb26806929c"
format_stamp: "Formatted at 2025-10-28 14:52:44 on dist-test-slave-0v9m"
I20251028 14:52:44.968171 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251028 14:52:44.969969 7267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:44.970471 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:52:44.970605 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "4cb3d6dfd43e4562bfd8ebb26806929c"
format_stamp: "Formatted at 2025-10-28 14:52:44 on dist-test-slave-0v9m"
I20251028 14:52:44.970767 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:44.980516 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:44.981129 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:44.983368 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:44.983505 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:44.983618 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:44.983707 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:45.012095 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33961
I20251028 14:52:45.012157 7336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33961 every 8 connection(s)
I20251028 14:52:45.014190 7337 heartbeater.cc:344] Connected to a master server at 127.0.104.62:33005
I20251028 14:52:45.014351 7337 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:45.014806 7337 heartbeater.cc:507] Master 127.0.104.62:33005 requested a full tablet report, sending...
I20251028 14:52:45.015786 7187 ts_manager.cc:194] Registered new tserver with Master: 4cb3d6dfd43e4562bfd8ebb26806929c (127.0.104.1:33961)
I20251028 14:52:45.016636 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003553831s
I20251028 14:52:45.017062 7187 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40844
I20251028 14:52:45.028669 7187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40854:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:52:45.044024 7302 tablet_service.cc:1505] Processing CreateTablet for tablet 40922a37c3d4414bbc0352b94643b812 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=264d7c2cf5da4f0ca8e9c6a7acc0ad0a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:45.044034 7301 tablet_service.cc:1505] Processing CreateTablet for tablet 069bc7c0f0f0452bb97e5183fc66f710 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=264d7c2cf5da4f0ca8e9c6a7acc0ad0a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:45.044692 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40922a37c3d4414bbc0352b94643b812. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:45.044950 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 069bc7c0f0f0452bb97e5183fc66f710. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:45.052676 7347 tablet_bootstrap.cc:492] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c: Bootstrap starting.
I20251028 14:52:45.054312 7347 tablet_bootstrap.cc:654] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:45.056435 7347 tablet_bootstrap.cc:492] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c: No bootstrap required, opened a new log
I20251028 14:52:45.056612 7347 ts_tablet_manager.cc:1403] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:45.057463 7347 raft_consensus.cc:359] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.057615 7347 raft_consensus.cc:385] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:45.057706 7347 raft_consensus.cc:740] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb3d6dfd43e4562bfd8ebb26806929c, State: Initialized, Role: FOLLOWER
I20251028 14:52:45.057937 7347 consensus_queue.cc:260] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.058106 7347 raft_consensus.cc:399] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:45.058187 7347 raft_consensus.cc:493] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:45.058291 7347 raft_consensus.cc:3060] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:45.060245 7347 raft_consensus.cc:515] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.060503 7347 leader_election.cc:304] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cb3d6dfd43e4562bfd8ebb26806929c; no voters:
I20251028 14:52:45.060989 7347 leader_election.cc:290] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:45.061201 7349 raft_consensus.cc:2804] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:45.061890 7337 heartbeater.cc:499] Master 127.0.104.62:33005 was elected leader, sending a full tablet report...
I20251028 14:52:45.062511 7347 ts_tablet_manager.cc:1434] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251028 14:52:45.063009 7347 tablet_bootstrap.cc:492] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c: Bootstrap starting.
I20251028 14:52:45.064404 7349 raft_consensus.cc:697] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 LEADER]: Becoming Leader. State: Replica: 4cb3d6dfd43e4562bfd8ebb26806929c, State: Running, Role: LEADER
I20251028 14:52:45.064783 7349 consensus_queue.cc:237] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.065670 7347 tablet_bootstrap.cc:654] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:45.068490 7347 tablet_bootstrap.cc:492] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c: No bootstrap required, opened a new log
I20251028 14:52:45.068740 7347 ts_tablet_manager.cc:1403] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251028 14:52:45.068641 7187 catalog_manager.cc:5649] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cb3d6dfd43e4562bfd8ebb26806929c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:45.069491 7347 raft_consensus.cc:359] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.069689 7347 raft_consensus.cc:385] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:45.069823 7347 raft_consensus.cc:740] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb3d6dfd43e4562bfd8ebb26806929c, State: Initialized, Role: FOLLOWER
I20251028 14:52:45.070097 7347 consensus_queue.cc:260] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.070243 7347 raft_consensus.cc:399] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:45.070338 7347 raft_consensus.cc:493] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:45.070441 7347 raft_consensus.cc:3060] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:45.072764 7347 raft_consensus.cc:515] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.073009 7347 leader_election.cc:304] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cb3d6dfd43e4562bfd8ebb26806929c; no voters:
I20251028 14:52:45.073213 7347 leader_election.cc:290] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:45.073572 7351 raft_consensus.cc:2804] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:45.073643 7347 ts_tablet_manager.cc:1434] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:52:45.073734 7351 raft_consensus.cc:697] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 LEADER]: Becoming Leader. State: Replica: 4cb3d6dfd43e4562bfd8ebb26806929c, State: Running, Role: LEADER
I20251028 14:52:45.074014 7351 consensus_queue.cc:237] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.076653 7187 catalog_manager.cc:5649] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cb3d6dfd43e4562bfd8ebb26806929c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:45.084666 7341 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:52:45.964063 7187 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54382:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:45.973973 7302 tablet_service.cc:1505] Processing CreateTablet for tablet ca4632867668466695fe4c247848b5ce (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=94dc325aed214c108e5a629ea2ad24d3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:45.974483 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca4632867668466695fe4c247848b5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:45.979511 7365 tablet_bootstrap.cc:492] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: Bootstrap starting.
I20251028 14:52:45.981228 7365 tablet_bootstrap.cc:654] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:45.983122 7365 tablet_bootstrap.cc:492] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: No bootstrap required, opened a new log
I20251028 14:52:45.983299 7365 ts_tablet_manager.cc:1403] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:45.984028 7365 raft_consensus.cc:359] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.984189 7365 raft_consensus.cc:385] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:45.984279 7365 raft_consensus.cc:740] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cb3d6dfd43e4562bfd8ebb26806929c, State: Initialized, Role: FOLLOWER
I20251028 14:52:45.984481 7365 consensus_queue.cc:260] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.984622 7365 raft_consensus.cc:399] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:45.984695 7365 raft_consensus.cc:493] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:45.984790 7365 raft_consensus.cc:3060] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:45.986735 7365 raft_consensus.cc:515] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.986979 7365 leader_election.cc:304] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4cb3d6dfd43e4562bfd8ebb26806929c; no voters:
I20251028 14:52:45.987471 7365 leader_election.cc:290] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:45.987680 7367 raft_consensus.cc:2804] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:45.988286 7365 ts_tablet_manager.cc:1434] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:52:45.988664 7367 raft_consensus.cc:697] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 LEADER]: Becoming Leader. State: Replica: 4cb3d6dfd43e4562bfd8ebb26806929c, State: Running, Role: LEADER
I20251028 14:52:45.988963 7367 consensus_queue.cc:237] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } }
I20251028 14:52:45.990258 7368 tablet_replica.cc:442] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } } }
I20251028 14:52:45.990321 7369 tablet_replica.cc:442] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: TxnStatusTablet state changed. Reason: New leader 4cb3d6dfd43e4562bfd8ebb26806929c. Latest consensus state: current_term: 1 leader_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } } }
I20251028 14:52:45.990432 7368 tablet_replica.cc:445] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:45.990478 7369 tablet_replica.cc:445] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:45.991532 7371 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:45.991706 7371 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:45.991627 7187 catalog_manager.cc:5649] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cb3d6dfd43e4562bfd8ebb26806929c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4cb3d6dfd43e4562bfd8ebb26806929c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33961 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:47.185525 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:47.185791 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:47.186131 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=0}) took 1007 ms. Trace:
I20251028 14:52:47.186213 7372 rpcz_store.cc:276] 1028 14:52:46.178742 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:46.178808 (+ 66us) service_pool.cc:225] Handling call
1028 14:52:47.186024 (+1007216us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:46.183273 (+ 0us) participant_op.cc:163] PREPARE: starting
1028 14:52:46.183307 (+ 34us) participant_op.cc:199] PREPARE: finished
1028 14:52:46.183338 (+ 31us) participant_op.cc:212] START. Timestamp: 1761663166183322
1028 14:52:46.183340 (+ 2us) op_driver.cc:348] REPLICATION: starting
1028 14:52:46.183489 (+ 149us) log.cc:844] Serialized 65 byte log entry
1028 14:52:46.183939 (+ 450us) op_driver.cc:464] REPLICATION: finished
1028 14:52:46.184043 (+ 104us) participant_op.cc:265] APPLY: starting
1028 14:52:46.184296 (+ 253us) participant_op.cc:270] APPLY: finished
1028 14:52:46.184328 (+ 32us) log.cc:844] Serialized 14 byte log entry
1028 14:52:46.184364 (+ 36us) participant_op.cc:94] Released txn lock
1028 14:52:46.184371 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
1028 14:52:46.184719 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:46.184824 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:46.184827 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:46.184828 (+ 1us) tablet.cc:660] Decoding operations
1028 14:52:46.184905 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:47.185727 (+1000822us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:47.185745 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:47.185754 (+ 9us) write_op.cc:454] Released schema lock
1028 14:52:47.185762 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":77,"txn-participant-registration.run_cpu_time_us":637,"txn-participant-registration.run_wall_time_us":4020,"child_traces":[["op",{"apply.queue_time_us":50,"apply.run_cpu_time_us":623,"apply.run_wall_time_us":623,"prepare.queue_time_us":97,"prepare.run_cpu_time_us":272,"prepare.run_wall_time_us":273,"replication_time_us":581,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":66}]]}]]}
W20251028 14:52:48.192318 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:48.192595 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:48.192920 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=1}) took 1002 ms. Trace:
I20251028 14:52:48.192999 7372 rpcz_store.cc:276] 1028 14:52:47.190761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:47.190841 (+ 80us) service_pool.cc:225] Handling call
1028 14:52:48.192854 (+1002013us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:47.191446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:47.191582 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:47.191585 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:47.191588 (+ 3us) tablet.cc:660] Decoding operations
1028 14:52:47.191672 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:48.192532 (+1000860us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:48.192547 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:48.192556 (+ 9us) write_op.cc:454] Released schema lock
1028 14:52:48.192565 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":78}]]}
W20251028 14:52:49.201303 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:49.201582 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:49.201898 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=2}) took 1002 ms. Trace:
I20251028 14:52:49.201977 7372 rpcz_store.cc:276] 1028 14:52:48.199579 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:48.199673 (+ 94us) service_pool.cc:225] Handling call
1028 14:52:49.201839 (+1002166us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:48.200314 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:48.200452 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:48.200454 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:52:48.200456 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:48.200538 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:49.201521 (+1000983us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:49.201535 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:49.201545 (+ 10us) write_op.cc:454] Released schema lock
1028 14:52:49.201553 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":104}]]}
W20251028 14:52:50.208846 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:50.209120 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:50.209504 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=3}) took 1002 ms. Trace:
I20251028 14:52:50.209584 7372 rpcz_store.cc:276] 1028 14:52:49.206757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:49.206913 (+ 156us) service_pool.cc:225] Handling call
1028 14:52:50.209439 (+1002526us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:49.207837 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:49.207975 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:49.207978 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:49.207979 (+ 1us) tablet.cc:660] Decoding operations
1028 14:52:49.208064 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:50.209056 (+1000992us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:50.209071 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:50.209080 (+ 9us) write_op.cc:454] Released schema lock
1028 14:52:50.209092 (+ 12us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":124}]]}
W20251028 14:52:51.220454 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:51.220706 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:51.221000 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=4}) took 1002 ms. Trace:
I20251028 14:52:51.221077 7372 rpcz_store.cc:276] 1028 14:52:50.218527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:50.218681 (+ 154us) service_pool.cc:225] Handling call
1028 14:52:51.220950 (+1002269us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:50.219537 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:50.219674 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:50.219677 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:50.219678 (+ 1us) tablet.cc:660] Decoding operations
1028 14:52:50.219765 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:51.220643 (+1000878us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:51.220660 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:51.220669 (+ 9us) write_op.cc:454] Released schema lock
1028 14:52:51.220678 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":142}]]}
W20251028 14:52:52.232959 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:52.233220 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:52.233537 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=5}) took 1002 ms. Trace:
I20251028 14:52:52.233613 7372 rpcz_store.cc:276] 1028 14:52:51.230820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:51.231027 (+ 207us) service_pool.cc:225] Handling call
1028 14:52:52.233475 (+1002448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:51.231779 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:51.231964 (+ 185us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:51.231966 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:52:51.231968 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:51.232082 (+ 114us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:52.233168 (+1001086us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:52.233183 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:52.233191 (+ 8us) write_op.cc:454] Released schema lock
1028 14:52:52.233198 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":110}]]}
W20251028 14:52:53.246762 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:53.247018 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:53.247316 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=6}) took 1002 ms. Trace:
I20251028 14:52:53.247390 7372 rpcz_store.cc:276] 1028 14:52:52.244419 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:52.244592 (+ 173us) service_pool.cc:225] Handling call
1028 14:52:53.247226 (+1002634us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:52.245762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:52.245912 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:52.245914 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:52:52.245916 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:52.246012 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:53.246966 (+1000954us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:53.246981 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:53.246987 (+ 6us) write_op.cc:454] Released schema lock
1028 14:52:53.246995 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":129}]]}
W20251028 14:52:54.259410 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:54.259656 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:54.259920 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=7}) took 1002 ms. Trace:
I20251028 14:52:54.259989 7372 rpcz_store.cc:276] 1028 14:52:53.257513 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:53.257637 (+ 124us) service_pool.cc:225] Handling call
1028 14:52:54.259869 (+1002232us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:53.258396 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:53.258530 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:53.258532 (+ 2us) write_op.cc:435] Acquired schema lock
1028 14:52:53.258534 (+ 2us) tablet.cc:660] Decoding operations
1028 14:52:53.258618 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:54.259606 (+1000988us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:54.259620 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:54.259627 (+ 7us) write_op.cc:454] Released schema lock
1028 14:52:54.259634 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":99}]]}
W20251028 14:52:55.272019 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:55.272331 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:55.272682 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=8}) took 1002 ms. Trace:
I20251028 14:52:55.272765 7372 rpcz_store.cc:276] 1028 14:52:54.270011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:54.270173 (+ 162us) service_pool.cc:225] Handling call
1028 14:52:55.272621 (+1002448us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:54.270985 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:54.271165 (+ 180us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:54.271168 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:54.271169 (+ 1us) tablet.cc:660] Decoding operations
1028 14:52:54.271281 (+ 112us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:55.272266 (+1000985us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:55.272285 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:55.272294 (+ 9us) write_op.cc:454] Released schema lock
1028 14:52:55.272305 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":127}]]}
W20251028 14:52:56.287884 7372 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20251028 14:52:56.288120 7372 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:41614: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20251028 14:52:56.288448 7372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41614 (ReqId={client: a6c2dd0dfeff482790957ff108c1c746, seq_no=5, attempt_no=9}) took 1002 ms. Trace:
I20251028 14:52:56.288558 7372 rpcz_store.cc:276] 1028 14:52:55.286024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:52:55.286166 (+ 142us) service_pool.cc:225] Handling call
1028 14:52:56.288391 (+1002225us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1028 14:52:55.286894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 069bc7c0f0f0452bb97e5183fc66f710
1028 14:52:55.287034 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
1028 14:52:55.287037 (+ 3us) write_op.cc:435] Acquired schema lock
1028 14:52:55.287038 (+ 1us) tablet.cc:660] Decoding operations
1028 14:52:55.287129 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
1028 14:52:56.288070 (+1000941us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
1028 14:52:56.288083 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
1028 14:52:56.288091 (+ 8us) write_op.cc:454] Released schema lock
1028 14:52:56.288097 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":94}]]}
W20251028 14:52:56.302714 7340 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 069bc7c0f0f0452bb97e5183fc66f710 after 11 attempt(s): Failed to write to server: (no server available): Write(tablet: 069bc7c0f0f0452bb97e5183fc66f710, num_ops: 1, num_attempts: 11) passed its deadline
I20251028 14:52:56.326071 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:52:56.338416 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:56.338762 416 tablet_replica.cc:333] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c: stopping tablet replica
I20251028 14:52:56.338972 416 raft_consensus.cc:2243] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:56.339185 416 raft_consensus.cc:2272] T ca4632867668466695fe4c247848b5ce P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:56.340082 416 tablet_replica.cc:333] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c: stopping tablet replica
I20251028 14:52:56.340214 416 raft_consensus.cc:2243] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:56.340389 416 raft_consensus.cc:2272] T 40922a37c3d4414bbc0352b94643b812 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:56.341097 416 tablet_replica.cc:333] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c: stopping tablet replica
I20251028 14:52:56.341254 416 raft_consensus.cc:2243] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:56.341446 416 raft_consensus.cc:2272] T 069bc7c0f0f0452bb97e5183fc66f710 P 4cb3d6dfd43e4562bfd8ebb26806929c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:56.356753 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:52:56.362422 416 master.cc:561] Master@127.0.104.62:33005 shutting down...
I20251028 14:52:56.371101 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:56.371359 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:56.371474 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P c47480ef006f47ce8d8192f3561f71ae: stopping tablet replica
I20251028 14:52:56.385198 416 master.cc:583] Master@127.0.104.62:33005 shutdown complete.
[ OK ] TxnOpDispatcherITest.BeginTxnLockRetry (11614 ms)
[ RUN ] TxnOpDispatcherITest.ErrorInParticipantRegistration
I20251028 14:52:56.415426 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:45957
I20251028 14:52:56.415963 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:56.419142 7395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:56.419129 7392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:56.419095 7393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:56.420066 416 server_base.cc:1047] running on GCE node
I20251028 14:52:56.420367 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:56.420428 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:56.420454 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663176420448 us; error 0 us; skew 500 ppm
I20251028 14:52:56.420625 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:56.421974 416 webserver.cc:492] Webserver started at http://127.0.104.62:39387/ using document root <none> and password file <none>
I20251028 14:52:56.422212 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:56.422281 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:56.422411 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:56.423029 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "db415aac6266482db04e6a4257fc5185"
format_stamp: "Formatted at 2025-10-28 14:52:56 on dist-test-slave-0v9m"
I20251028 14:52:56.425688 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:56.427397 7400 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:56.427798 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:52:56.427908 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "db415aac6266482db04e6a4257fc5185"
format_stamp: "Formatted at 2025-10-28 14:52:56 on dist-test-slave-0v9m"
I20251028 14:52:56.428021 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:56.452534 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:56.453096 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:56.468367 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:45957
I20251028 14:52:56.468390 7461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:45957 every 8 connection(s)
I20251028 14:52:56.470005 7462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:56.474249 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185: Bootstrap starting.
I20251028 14:52:56.475644 7462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:56.477303 7462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185: No bootstrap required, opened a new log
I20251028 14:52:56.477983 7462 raft_consensus.cc:359] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db415aac6266482db04e6a4257fc5185" member_type: VOTER }
I20251028 14:52:56.478133 7462 raft_consensus.cc:385] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:56.478171 7462 raft_consensus.cc:740] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db415aac6266482db04e6a4257fc5185, State: Initialized, Role: FOLLOWER
I20251028 14:52:56.478380 7462 consensus_queue.cc:260] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db415aac6266482db04e6a4257fc5185" member_type: VOTER }
I20251028 14:52:56.478499 7462 raft_consensus.cc:399] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:56.478538 7462 raft_consensus.cc:493] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:56.478587 7462 raft_consensus.cc:3060] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:56.480646 7462 raft_consensus.cc:515] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db415aac6266482db04e6a4257fc5185" member_type: VOTER }
I20251028 14:52:56.480880 7462 leader_election.cc:304] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db415aac6266482db04e6a4257fc5185; no voters:
I20251028 14:52:56.481413 7462 leader_election.cc:290] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:56.481537 7465 raft_consensus.cc:2804] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:56.481973 7465 raft_consensus.cc:697] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 1 LEADER]: Becoming Leader. State: Replica: db415aac6266482db04e6a4257fc5185, State: Running, Role: LEADER
I20251028 14:52:56.482210 7465 consensus_queue.cc:237] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db415aac6266482db04e6a4257fc5185" member_type: VOTER }
I20251028 14:52:56.482280 7462 sys_catalog.cc:565] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:56.482980 7466 sys_catalog.cc:455] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db415aac6266482db04e6a4257fc5185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db415aac6266482db04e6a4257fc5185" member_type: VOTER } }
I20251028 14:52:56.482969 7467 sys_catalog.cc:455] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db415aac6266482db04e6a4257fc5185. Latest consensus state: current_term: 1 leader_uuid: "db415aac6266482db04e6a4257fc5185" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db415aac6266482db04e6a4257fc5185" member_type: VOTER } }
I20251028 14:52:56.483167 7467 sys_catalog.cc:458] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:56.484208 7466 sys_catalog.cc:458] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:56.484370 7470 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:56.485687 7470 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:56.487208 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:52:56.488983 7470 catalog_manager.cc:1357] Generated new cluster ID: 46de5bef96a640fea05c3dfcee8a6eb2
I20251028 14:52:56.489084 7470 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:56.504392 7470 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:56.505270 7470 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:56.524464 7470 catalog_manager.cc:6022] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185: Generated new TSK 0
I20251028 14:52:56.524758 7470 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:56.536675 7417 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43542:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:56.552667 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:56.555729 7489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:56.555805 7490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:56.556216 7492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:56.556571 416 server_base.cc:1047] running on GCE node
I20251028 14:52:56.556855 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:56.556913 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:56.556938 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663176556932 us; error 0 us; skew 500 ppm
I20251028 14:52:56.557067 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:56.558408 416 webserver.cc:492] Webserver started at http://127.0.104.1:46063/ using document root <none> and password file <none>
I20251028 14:52:56.558645 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:56.558717 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:56.558838 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:56.559443 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "c888011ff02b4cee8515036c846ed007"
format_stamp: "Formatted at 2025-10-28 14:52:56 on dist-test-slave-0v9m"
I20251028 14:52:56.561998 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251028 14:52:56.563786 7497 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:56.564160 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251028 14:52:56.564272 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "c888011ff02b4cee8515036c846ed007"
format_stamp: "Formatted at 2025-10-28 14:52:56 on dist-test-slave-0v9m"
I20251028 14:52:56.564404 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:56.588326 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:56.588922 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:56.590893 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:56.591090 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:56.591233 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:56.591372 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:56.613457 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46323
I20251028 14:52:56.613479 7566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46323 every 8 connection(s)
I20251028 14:52:56.615379 7567 heartbeater.cc:344] Connected to a master server at 127.0.104.62:45957
I20251028 14:52:56.615548 7567 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:56.616015 7567 heartbeater.cc:507] Master 127.0.104.62:45957 requested a full tablet report, sending...
I20251028 14:52:56.617100 7417 ts_manager.cc:194] Registered new tserver with Master: c888011ff02b4cee8515036c846ed007 (127.0.104.1:46323)
I20251028 14:52:56.617880 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003490558s
I20251028 14:52:56.618551 7417 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43548
I20251028 14:52:56.630726 7417 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43566:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:52:56.647945 7530 tablet_service.cc:1505] Processing CreateTablet for tablet c0462113cc8f46b39c50fba463e79175 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=2fbb2dca0888470da660cb8af2870d3a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:56.647972 7529 tablet_service.cc:1505] Processing CreateTablet for tablet 74c9525d0a8d443da927b52f922935c5 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=2fbb2dca0888470da660cb8af2870d3a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:56.648484 7530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0462113cc8f46b39c50fba463e79175. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:56.648663 7529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74c9525d0a8d443da927b52f922935c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:56.653851 7577 tablet_bootstrap.cc:492] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007: Bootstrap starting.
I20251028 14:52:56.655392 7577 tablet_bootstrap.cc:654] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:56.657133 7577 tablet_bootstrap.cc:492] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007: No bootstrap required, opened a new log
I20251028 14:52:56.657305 7577 ts_tablet_manager.cc:1403] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:56.657917 7577 raft_consensus.cc:359] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.658054 7577 raft_consensus.cc:385] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:56.658113 7577 raft_consensus.cc:740] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c888011ff02b4cee8515036c846ed007, State: Initialized, Role: FOLLOWER
I20251028 14:52:56.658349 7577 consensus_queue.cc:260] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.658471 7577 raft_consensus.cc:399] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:56.658535 7577 raft_consensus.cc:493] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:56.658617 7577 raft_consensus.cc:3060] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:56.660629 7577 raft_consensus.cc:515] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.660876 7577 leader_election.cc:304] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c888011ff02b4cee8515036c846ed007; no voters:
I20251028 14:52:56.661353 7577 leader_election.cc:290] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:56.661583 7579 raft_consensus.cc:2804] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:56.662113 7567 heartbeater.cc:499] Master 127.0.104.62:45957 was elected leader, sending a full tablet report...
I20251028 14:52:56.662178 7577 ts_tablet_manager.cc:1434] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20251028 14:52:56.662446 7579 raft_consensus.cc:697] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 1 LEADER]: Becoming Leader. State: Replica: c888011ff02b4cee8515036c846ed007, State: Running, Role: LEADER
I20251028 14:52:56.662554 7577 tablet_bootstrap.cc:492] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007: Bootstrap starting.
I20251028 14:52:56.662739 7579 consensus_queue.cc:237] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.664448 7577 tablet_bootstrap.cc:654] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:56.665589 7417 catalog_manager.cc:5649] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 reported cstate change: term changed from 0 to 1, leader changed from <none> to c888011ff02b4cee8515036c846ed007 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c888011ff02b4cee8515036c846ed007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:56.668381 7577 tablet_bootstrap.cc:492] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007: No bootstrap required, opened a new log
I20251028 14:52:56.668572 7577 ts_tablet_manager.cc:1403] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251028 14:52:56.669296 7577 raft_consensus.cc:359] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.669503 7577 raft_consensus.cc:385] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:56.669598 7577 raft_consensus.cc:740] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c888011ff02b4cee8515036c846ed007, State: Initialized, Role: FOLLOWER
I20251028 14:52:56.669822 7577 consensus_queue.cc:260] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.669945 7577 raft_consensus.cc:399] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:56.670040 7577 raft_consensus.cc:493] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:56.670137 7577 raft_consensus.cc:3060] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:56.672101 7577 raft_consensus.cc:515] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.672293 7577 leader_election.cc:304] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c888011ff02b4cee8515036c846ed007; no voters:
I20251028 14:52:56.672474 7577 leader_election.cc:290] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:56.672552 7579 raft_consensus.cc:2804] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:56.672708 7579 raft_consensus.cc:697] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 1 LEADER]: Becoming Leader. State: Replica: c888011ff02b4cee8515036c846ed007, State: Running, Role: LEADER
I20251028 14:52:56.672927 7577 ts_tablet_manager.cc:1434] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251028 14:52:56.672983 7579 consensus_queue.cc:237] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:56.675058 7416 catalog_manager.cc:5649] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 reported cstate change: term changed from 0 to 1, leader changed from <none> to c888011ff02b4cee8515036c846ed007 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c888011ff02b4cee8515036c846ed007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:56.687451 7570 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:52:57.552676 7416 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43578:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:57.563056 7529 tablet_service.cc:1505] Processing CreateTablet for tablet d21ce756251944ffbcc17b5dace3842f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7011ca9a9eb4471ba953600b6974d4b7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:57.563553 7529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d21ce756251944ffbcc17b5dace3842f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:57.568919 7595 tablet_bootstrap.cc:492] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: Bootstrap starting.
I20251028 14:52:57.570967 7595 tablet_bootstrap.cc:654] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:57.573748 7595 tablet_bootstrap.cc:492] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: No bootstrap required, opened a new log
I20251028 14:52:57.574254 7595 ts_tablet_manager.cc:1403] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:52:57.575793 7595 raft_consensus.cc:359] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:57.576040 7595 raft_consensus.cc:385] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:57.576143 7595 raft_consensus.cc:740] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c888011ff02b4cee8515036c846ed007, State: Initialized, Role: FOLLOWER
I20251028 14:52:57.576387 7595 consensus_queue.cc:260] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:57.576536 7595 raft_consensus.cc:399] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:57.576586 7595 raft_consensus.cc:493] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:57.576699 7595 raft_consensus.cc:3060] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:57.578714 7595 raft_consensus.cc:515] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:57.578981 7595 leader_election.cc:304] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c888011ff02b4cee8515036c846ed007; no voters:
I20251028 14:52:57.579638 7595 leader_election.cc:290] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:57.579805 7597 raft_consensus.cc:2804] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:57.580507 7595 ts_tablet_manager.cc:1434] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251028 14:52:57.580858 7597 raft_consensus.cc:697] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 1 LEADER]: Becoming Leader. State: Replica: c888011ff02b4cee8515036c846ed007, State: Running, Role: LEADER
I20251028 14:52:57.581537 7597 consensus_queue.cc:237] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } }
I20251028 14:52:57.582819 7598 tablet_replica.cc:442] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c888011ff02b4cee8515036c846ed007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } } }
I20251028 14:52:57.583060 7598 tablet_replica.cc:445] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:57.583141 7599 tablet_replica.cc:442] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: TxnStatusTablet state changed. Reason: New leader c888011ff02b4cee8515036c846ed007. Latest consensus state: current_term: 1 leader_uuid: "c888011ff02b4cee8515036c846ed007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } } }
I20251028 14:52:57.583374 7599 tablet_replica.cc:445] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:57.584510 7601 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:57.584721 7601 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:52:57.584911 7416 catalog_manager.cc:5649] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 reported cstate change: term changed from 0 to 1, leader changed from <none> to c888011ff02b4cee8515036c846ed007 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c888011ff02b4cee8515036c846ed007" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c888011ff02b4cee8515036c846ed007" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46323 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:57.708076 7608 tablet_replica.cc:1306] Invalid argument: Failed to write to server: c888011ff02b4cee8515036c846ed007 (127.0.104.1:46323): transaction ID 10 not found, current highest txn ID: 0: cancelling pending write operations [suppressed 1 similar messages]
W20251028 14:52:57.708241 7608 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60782: Invalid argument: Failed to write to server: c888011ff02b4cee8515036c846ed007 (127.0.104.1:46323): transaction ID 10 not found, current highest txn ID: 0
W20251028 14:52:57.709115 7570 batcher.cc:441] Invalid argument: Failed to write batch of 1 ops to tablet c0462113cc8f46b39c50fba463e79175 after 1 attempt(s): Failed to write to server: c888011ff02b4cee8515036c846ed007 (127.0.104.1:46323): Failed to write to server: c888011ff02b4cee8515036c846ed007 (127.0.104.1:46323): transaction ID 10 not found, current highest txn ID: 0
W20251028 14:52:57.750059 7608 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60782: Illegal state: Failed to write to server: c888011ff02b4cee8515036c846ed007 (127.0.104.1:46323): transaction ID 1 not open: COMMITTED
I20251028 14:52:57.753813 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:52:57.766165 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:57.768785 416 tablet_replica.cc:333] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007: stopping tablet replica
I20251028 14:52:57.769326 416 raft_consensus.cc:2243] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:57.769920 416 raft_consensus.cc:2272] T c0462113cc8f46b39c50fba463e79175 P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:57.770761 416 tablet_replica.cc:333] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007: stopping tablet replica
I20251028 14:52:57.770932 416 raft_consensus.cc:2243] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:57.771121 416 raft_consensus.cc:2272] T d21ce756251944ffbcc17b5dace3842f P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:57.772955 416 tablet_replica.cc:333] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007: stopping tablet replica
I20251028 14:52:57.773191 416 raft_consensus.cc:2243] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:57.773674 416 raft_consensus.cc:2272] T 74c9525d0a8d443da927b52f922935c5 P c888011ff02b4cee8515036c846ed007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:57.781338 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:52:57.787844 416 master.cc:561] Master@127.0.104.62:45957 shutting down...
I20251028 14:52:57.800590 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:57.800895 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:57.801031 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P db415aac6266482db04e6a4257fc5185: stopping tablet replica
I20251028 14:52:57.816917 416 master.cc:583] Master@127.0.104.62:45957 shutdown complete.
[ OK ] TxnOpDispatcherITest.ErrorInParticipantRegistration (1439 ms)
[ RUN ] TxnOpDispatcherITest.ErrorInProcessingWriteOp
I20251028 14:52:57.854979 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:41825
I20251028 14:52:57.855562 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:57.858749 7617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:57.858749 7618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:57.859213 7620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:57.859658 416 server_base.cc:1047] running on GCE node
I20251028 14:52:57.860031 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:57.860090 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:57.860127 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663177860118 us; error 0 us; skew 500 ppm
I20251028 14:52:57.860291 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:57.861670 416 webserver.cc:492] Webserver started at http://127.0.104.62:39839/ using document root <none> and password file <none>
I20251028 14:52:57.861896 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:57.861963 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:57.862078 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:57.862761 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "3e3d7970b88a4ad1989c922d43efbeec"
format_stamp: "Formatted at 2025-10-28 14:52:57 on dist-test-slave-0v9m"
I20251028 14:52:57.865226 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:57.867048 7625 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:57.867435 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:52:57.867552 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "3e3d7970b88a4ad1989c922d43efbeec"
format_stamp: "Formatted at 2025-10-28 14:52:57 on dist-test-slave-0v9m"
I20251028 14:52:57.867681 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:57.874715 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:57.875314 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:57.893206 7686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41825 every 8 connection(s)
I20251028 14:52:57.893206 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41825
I20251028 14:52:57.894688 7687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:57.899927 7687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec: Bootstrap starting.
I20251028 14:52:57.901679 7687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:57.903736 7687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec: No bootstrap required, opened a new log
I20251028 14:52:57.904808 7687 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3d7970b88a4ad1989c922d43efbeec" member_type: VOTER }
I20251028 14:52:57.905050 7687 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:57.905112 7687 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3d7970b88a4ad1989c922d43efbeec, State: Initialized, Role: FOLLOWER
I20251028 14:52:57.905493 7687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3d7970b88a4ad1989c922d43efbeec" member_type: VOTER }
I20251028 14:52:57.905730 7687 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:57.905802 7687 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:57.905884 7687 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:57.907847 7687 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3d7970b88a4ad1989c922d43efbeec" member_type: VOTER }
I20251028 14:52:57.908090 7687 leader_election.cc:304] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e3d7970b88a4ad1989c922d43efbeec; no voters:
I20251028 14:52:57.908933 7687 leader_election.cc:290] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:57.909067 7690 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:57.909729 7690 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 1 LEADER]: Becoming Leader. State: Replica: 3e3d7970b88a4ad1989c922d43efbeec, State: Running, Role: LEADER
I20251028 14:52:57.910053 7690 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3d7970b88a4ad1989c922d43efbeec" member_type: VOTER }
I20251028 14:52:57.910223 7687 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:52:57.911022 7691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e3d7970b88a4ad1989c922d43efbeec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3d7970b88a4ad1989c922d43efbeec" member_type: VOTER } }
I20251028 14:52:57.911074 7692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3d7970b88a4ad1989c922d43efbeec. Latest consensus state: current_term: 1 leader_uuid: "3e3d7970b88a4ad1989c922d43efbeec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3d7970b88a4ad1989c922d43efbeec" member_type: VOTER } }
I20251028 14:52:57.911322 7692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:57.911850 7691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [sys.catalog]: This master's current role is: LEADER
I20251028 14:52:57.912242 7694 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:52:57.913836 7694 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:52:57.916417 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:52:57.917631 7694 catalog_manager.cc:1357] Generated new cluster ID: 23d642e6e667472c9d179707b8a3b59f
I20251028 14:52:57.917725 7694 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:52:57.945437 7694 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:52:57.946178 7694 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:52:57.957237 7694 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec: Generated new TSK 0
I20251028 14:52:57.957525 7694 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:52:57.966339 7642 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38970:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:57.982304 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:57.985570 7715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:57.985639 7714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:57.986465 7717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:57.986542 416 server_base.cc:1047] running on GCE node
I20251028 14:52:57.986925 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:57.986991 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:57.987020 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663177987013 us; error 0 us; skew 500 ppm
I20251028 14:52:57.987138 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:57.988503 416 webserver.cc:492] Webserver started at http://127.0.104.1:37459/ using document root <none> and password file <none>
I20251028 14:52:57.988756 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:57.988821 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:57.988920 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:57.989619 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "3eeff0e5da904b748f940d2fd3e53bd8"
format_stamp: "Formatted at 2025-10-28 14:52:57 on dist-test-slave-0v9m"
I20251028 14:52:57.992328 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:52:57.994535 7722 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:57.995034 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251028 14:52:57.995147 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "3eeff0e5da904b748f940d2fd3e53bd8"
format_stamp: "Formatted at 2025-10-28 14:52:57 on dist-test-slave-0v9m"
I20251028 14:52:57.995272 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:52:58.003826 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:52:58.004431 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:52:58.006245 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:52:58.006353 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:58.006445 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:52:58.006501 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:52:58.028834 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:44153
I20251028 14:52:58.028874 7791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:44153 every 8 connection(s)
I20251028 14:52:58.030668 7792 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41825
I20251028 14:52:58.030817 7792 heartbeater.cc:461] Registering TS with master...
I20251028 14:52:58.031162 7792 heartbeater.cc:507] Master 127.0.104.62:41825 requested a full tablet report, sending...
I20251028 14:52:58.032120 7642 ts_manager.cc:194] Registered new tserver with Master: 3eeff0e5da904b748f940d2fd3e53bd8 (127.0.104.1:44153)
I20251028 14:52:58.032193 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002375595s
I20251028 14:52:58.033612 7642 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38978
I20251028 14:52:58.045140 7642 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38994:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:52:58.059958 7757 tablet_service.cc:1505] Processing CreateTablet for tablet 54a4636cf01248bd9ca01d3b25bfccbf (DEFAULT_TABLE table=txn_op_dispatcher_test [id=fdaa952bc4fc4968be0eb16555c3c442]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:58.060024 7756 tablet_service.cc:1505] Processing CreateTablet for tablet 727d1bac6f3141a48fa9e76de6a55754 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=fdaa952bc4fc4968be0eb16555c3c442]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:52:58.060492 7757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54a4636cf01248bd9ca01d3b25bfccbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:58.060708 7756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 727d1bac6f3141a48fa9e76de6a55754. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:58.065773 7802 tablet_bootstrap.cc:492] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8: Bootstrap starting.
I20251028 14:52:58.067242 7802 tablet_bootstrap.cc:654] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:58.069547 7802 tablet_bootstrap.cc:492] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8: No bootstrap required, opened a new log
I20251028 14:52:58.069680 7802 ts_tablet_manager.cc:1403] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:52:58.070241 7802 raft_consensus.cc:359] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.070416 7802 raft_consensus.cc:385] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:58.070456 7802 raft_consensus.cc:740] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eeff0e5da904b748f940d2fd3e53bd8, State: Initialized, Role: FOLLOWER
I20251028 14:52:58.070631 7802 consensus_queue.cc:260] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.070753 7802 raft_consensus.cc:399] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:58.070799 7802 raft_consensus.cc:493] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:58.070851 7802 raft_consensus.cc:3060] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:58.072834 7802 raft_consensus.cc:515] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.073071 7802 leader_election.cc:304] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eeff0e5da904b748f940d2fd3e53bd8; no voters:
I20251028 14:52:58.073729 7802 leader_election.cc:290] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:58.073892 7804 raft_consensus.cc:2804] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:58.074795 7792 heartbeater.cc:499] Master 127.0.104.62:41825 was elected leader, sending a full tablet report...
I20251028 14:52:58.074895 7802 ts_tablet_manager.cc:1434] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:52:58.075171 7804 raft_consensus.cc:697] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 3eeff0e5da904b748f940d2fd3e53bd8, State: Running, Role: LEADER
I20251028 14:52:58.075174 7802 tablet_bootstrap.cc:492] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8: Bootstrap starting.
I20251028 14:52:58.075495 7804 consensus_queue.cc:237] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.076958 7802 tablet_bootstrap.cc:654] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:58.078213 7642 catalog_manager.cc:5649] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eeff0e5da904b748f940d2fd3e53bd8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:58.078914 7802 tablet_bootstrap.cc:492] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8: No bootstrap required, opened a new log
I20251028 14:52:58.079069 7802 ts_tablet_manager.cc:1403] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251028 14:52:58.079797 7802 raft_consensus.cc:359] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.079986 7802 raft_consensus.cc:385] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:58.080060 7802 raft_consensus.cc:740] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eeff0e5da904b748f940d2fd3e53bd8, State: Initialized, Role: FOLLOWER
I20251028 14:52:58.080323 7802 consensus_queue.cc:260] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.080521 7802 raft_consensus.cc:399] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:58.080628 7802 raft_consensus.cc:493] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:58.080739 7802 raft_consensus.cc:3060] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:58.083058 7802 raft_consensus.cc:515] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.083590 7802 leader_election.cc:304] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eeff0e5da904b748f940d2fd3e53bd8; no voters:
I20251028 14:52:58.083982 7802 leader_election.cc:290] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:58.084038 7804 raft_consensus.cc:2804] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:58.084285 7804 raft_consensus.cc:697] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 3eeff0e5da904b748f940d2fd3e53bd8, State: Running, Role: LEADER
I20251028 14:52:58.084614 7802 ts_tablet_manager.cc:1434] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20251028 14:52:58.084601 7804 consensus_queue.cc:237] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:58.087934 7642 catalog_manager.cc:5649] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eeff0e5da904b748f940d2fd3e53bd8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } health_report { overall_health: HEALTHY } } }
W20251028 14:52:58.100932 7796 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:52:58.981817 7642 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38998:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:52:58.991665 7757 tablet_service.cc:1505] Processing CreateTablet for tablet e659f6a2e30c4731883a32e9109be97e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d3c2ae28913d4bcc989b901daeb16222]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:52:58.992121 7757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e659f6a2e30c4731883a32e9109be97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:52:58.997354 7820 tablet_bootstrap.cc:492] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: Bootstrap starting.
I20251028 14:52:58.999513 7820 tablet_bootstrap.cc:654] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: Neither blocks nor log segments found. Creating new log.
I20251028 14:52:59.001312 7820 tablet_bootstrap.cc:492] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: No bootstrap required, opened a new log
I20251028 14:52:59.001495 7820 ts_tablet_manager.cc:1403] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20251028 14:52:59.002177 7820 raft_consensus.cc:359] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:59.002322 7820 raft_consensus.cc:385] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:52:59.002399 7820 raft_consensus.cc:740] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eeff0e5da904b748f940d2fd3e53bd8, State: Initialized, Role: FOLLOWER
I20251028 14:52:59.002662 7820 consensus_queue.cc:260] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:59.002801 7820 raft_consensus.cc:399] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:52:59.002877 7820 raft_consensus.cc:493] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:52:59.002983 7820 raft_consensus.cc:3060] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:52:59.005595 7820 raft_consensus.cc:515] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:59.005960 7820 leader_election.cc:304] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eeff0e5da904b748f940d2fd3e53bd8; no voters:
I20251028 14:52:59.006577 7820 leader_election.cc:290] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:52:59.007002 7822 raft_consensus.cc:2804] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:52:59.007474 7820 ts_tablet_manager.cc:1434] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20251028 14:52:59.008023 7822 raft_consensus.cc:697] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 LEADER]: Becoming Leader. State: Replica: 3eeff0e5da904b748f940d2fd3e53bd8, State: Running, Role: LEADER
I20251028 14:52:59.008394 7822 consensus_queue.cc:237] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } }
I20251028 14:52:59.009632 7823 tablet_replica.cc:442] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } } }
I20251028 14:52:59.009799 7824 tablet_replica.cc:442] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: TxnStatusTablet state changed. Reason: New leader 3eeff0e5da904b748f940d2fd3e53bd8. Latest consensus state: current_term: 1 leader_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } } }
I20251028 14:52:59.009941 7823 tablet_replica.cc:445] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:59.010041 7824 tablet_replica.cc:445] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:52:59.011154 7642 catalog_manager.cc:5649] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3eeff0e5da904b748f940d2fd3e53bd8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eeff0e5da904b748f940d2fd3e53bd8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44153 } health_report { overall_health: HEALTHY } } }
I20251028 14:52:59.011937 7826 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:52:59.012112 7826 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251028 14:52:59.816098 7831 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:50856: Not implemented: transactions may only insert
W20251028 14:52:59.816816 7796 batcher.cc:441] Not implemented: Failed to write batch of 1 ops to tablet 54a4636cf01248bd9ca01d3b25bfccbf after 1 attempt(s): Failed to write to server: 3eeff0e5da904b748f940d2fd3e53bd8 (127.0.104.1:44153): transactions may only insert
I20251028 14:52:59.898353 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:52:59.909335 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:52:59.910044 416 tablet_replica.cc:333] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8: stopping tablet replica
I20251028 14:52:59.910337 416 raft_consensus.cc:2243] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:59.910759 416 raft_consensus.cc:2272] T e659f6a2e30c4731883a32e9109be97e P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:59.912034 416 tablet_replica.cc:333] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8: stopping tablet replica
I20251028 14:52:59.912222 416 raft_consensus.cc:2243] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:59.912434 416 raft_consensus.cc:2272] T 727d1bac6f3141a48fa9e76de6a55754 P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:59.913463 416 tablet_replica.cc:333] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8: stopping tablet replica
I20251028 14:52:59.913654 416 raft_consensus.cc:2243] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:59.913831 416 raft_consensus.cc:2272] T 54a4636cf01248bd9ca01d3b25bfccbf P 3eeff0e5da904b748f940d2fd3e53bd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:59.930101 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:52:59.937945 416 master.cc:561] Master@127.0.104.62:41825 shutting down...
I20251028 14:52:59.951510 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:52:59.951797 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:52:59.951941 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3e3d7970b88a4ad1989c922d43efbeec: stopping tablet replica
I20251028 14:52:59.979629 416 master.cc:583] Master@127.0.104.62:41825 shutdown complete.
[ OK ] TxnOpDispatcherITest.ErrorInProcessingWriteOp (2136 ms)
[ RUN ] TxnOpDispatcherITest.NoPendingWriteOps
I20251028 14:52:59.990943 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:39767
I20251028 14:52:59.991497 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:52:59.994117 7847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:59.994422 7846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:52:59.994763 7849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:52:59.995172 416 server_base.cc:1047] running on GCE node
I20251028 14:52:59.995482 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:52:59.995533 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:52:59.995563 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663179995558 us; error 0 us; skew 500 ppm
I20251028 14:52:59.995715 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:52:59.996979 416 webserver.cc:492] Webserver started at http://127.0.104.62:33417/ using document root <none> and password file <none>
I20251028 14:52:59.997201 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:52:59.997288 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:52:59.997422 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:52:59.998036 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c"
format_stamp: "Formatted at 2025-10-28 14:52:59 on dist-test-slave-0v9m"
I20251028 14:53:00.000536 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251028 14:53:00.002197 7854 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:00.002568 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:53:00.002660 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c"
format_stamp: "Formatted at 2025-10-28 14:52:59 on dist-test-slave-0v9m"
I20251028 14:53:00.002787 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:00.021888 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:00.022531 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:00.041090 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:39767
I20251028 14:53:00.041103 7915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:39767 every 8 connection(s)
I20251028 14:53:00.042723 7916 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:00.047072 7916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c: Bootstrap starting.
I20251028 14:53:00.048591 7916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:00.050333 7916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c: No bootstrap required, opened a new log
I20251028 14:53:00.050968 7916 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" member_type: VOTER }
I20251028 14:53:00.051110 7916 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:00.051146 7916 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8ec07b2e3b4f43ae1a9f59deebe85c, State: Initialized, Role: FOLLOWER
I20251028 14:53:00.051347 7916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" member_type: VOTER }
I20251028 14:53:00.051466 7916 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:00.051518 7916 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:00.051589 7916 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:00.053575 7916 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" member_type: VOTER }
I20251028 14:53:00.053773 7916 leader_election.cc:304] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f8ec07b2e3b4f43ae1a9f59deebe85c; no voters:
I20251028 14:53:00.054224 7916 leader_election.cc:290] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:00.054343 7919 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:00.054979 7919 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 1 LEADER]: Becoming Leader. State: Replica: 4f8ec07b2e3b4f43ae1a9f59deebe85c, State: Running, Role: LEADER
I20251028 14:53:00.055369 7916 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:00.055292 7919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" member_type: VOTER }
I20251028 14:53:00.056382 7920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" member_type: VOTER } }
I20251028 14:53:00.056841 7920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:00.061816 7921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f8ec07b2e3b4f43ae1a9f59deebe85c. Latest consensus state: current_term: 1 leader_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8ec07b2e3b4f43ae1a9f59deebe85c" member_type: VOTER } }
I20251028 14:53:00.062646 7921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:00.064827 7924 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:00.066793 7924 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:00.068976 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:00.074270 7924 catalog_manager.cc:1357] Generated new cluster ID: 5e456ea692bf41748c69e2058bb90a88
I20251028 14:53:00.074424 7924 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:00.114310 7924 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:00.115149 7924 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:00.131763 7924 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c: Generated new TSK 0
I20251028 14:53:00.132192 7924 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:00.135617 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:00.141002 7944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:00.141512 7945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:00.141732 7947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:00.141812 7871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44710:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:00.143111 416 server_base.cc:1047] running on GCE node
I20251028 14:53:00.143699 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:00.143790 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:00.143829 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663180143820 us; error 0 us; skew 500 ppm
I20251028 14:53:00.144011 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:00.145947 416 webserver.cc:492] Webserver started at http://127.0.104.1:39149/ using document root <none> and password file <none>
I20251028 14:53:00.146387 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:00.146584 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:00.146965 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:00.148097 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "940100f844c94726b682a02d1e16b69d"
format_stamp: "Formatted at 2025-10-28 14:53:00 on dist-test-slave-0v9m"
I20251028 14:53:00.151633 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251028 14:53:00.153820 7952 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:00.154266 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:53:00.154402 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "940100f844c94726b682a02d1e16b69d"
format_stamp: "Formatted at 2025-10-28 14:53:00 on dist-test-slave-0v9m"
I20251028 14:53:00.154551 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:00.176591 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:00.177440 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:00.180006 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:00.180150 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:00.180272 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:00.180387 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:00.231948 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40325
I20251028 14:53:00.232012 8021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40325 every 8 connection(s)
I20251028 14:53:00.234771 8022 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39767
I20251028 14:53:00.234925 8022 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:00.235314 8022 heartbeater.cc:507] Master 127.0.104.62:39767 requested a full tablet report, sending...
I20251028 14:53:00.236372 7871 ts_manager.cc:194] Registered new tserver with Master: 940100f844c94726b682a02d1e16b69d (127.0.104.1:40325)
I20251028 14:53:00.236841 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003454341s
I20251028 14:53:00.237635 7871 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44726
I20251028 14:53:00.258874 7871 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44740:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:00.276608 7987 tablet_service.cc:1505] Processing CreateTablet for tablet c7f565faca914c18bd0d987f83edabac (DEFAULT_TABLE table=txn_op_dispatcher_test [id=e97eaa226937498a8db0005a6af34c71]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:00.276677 7986 tablet_service.cc:1505] Processing CreateTablet for tablet 8cc26548c17f4e18963244db9442355c (DEFAULT_TABLE table=txn_op_dispatcher_test [id=e97eaa226937498a8db0005a6af34c71]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:00.277091 7987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7f565faca914c18bd0d987f83edabac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:00.277343 7986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cc26548c17f4e18963244db9442355c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:00.284595 8032 tablet_bootstrap.cc:492] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d: Bootstrap starting.
I20251028 14:53:00.286404 8032 tablet_bootstrap.cc:654] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:00.288734 8032 tablet_bootstrap.cc:492] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d: No bootstrap required, opened a new log
I20251028 14:53:00.288930 8032 ts_tablet_manager.cc:1403] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:00.289767 8032 raft_consensus.cc:359] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.289952 8032 raft_consensus.cc:385] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:00.290007 8032 raft_consensus.cc:740] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940100f844c94726b682a02d1e16b69d, State: Initialized, Role: FOLLOWER
I20251028 14:53:00.290216 8032 consensus_queue.cc:260] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.290380 8032 raft_consensus.cc:399] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:00.290438 8032 raft_consensus.cc:493] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:00.290517 8032 raft_consensus.cc:3060] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:00.293462 8032 raft_consensus.cc:515] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.293768 8032 leader_election.cc:304] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 940100f844c94726b682a02d1e16b69d; no voters:
I20251028 14:53:00.294422 8032 leader_election.cc:290] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:00.294773 8034 raft_consensus.cc:2804] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:00.295768 8022 heartbeater.cc:499] Master 127.0.104.62:39767 was elected leader, sending a full tablet report...
I20251028 14:53:00.295965 8034 raft_consensus.cc:697] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 1 LEADER]: Becoming Leader. State: Replica: 940100f844c94726b682a02d1e16b69d, State: Running, Role: LEADER
I20251028 14:53:00.296200 8034 consensus_queue.cc:237] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.296878 8032 ts_tablet_manager.cc:1434] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20251028 14:53:00.297245 8032 tablet_bootstrap.cc:492] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d: Bootstrap starting.
I20251028 14:53:00.298712 7871 catalog_manager.cc:5649] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d reported cstate change: term changed from 0 to 1, leader changed from <none> to 940100f844c94726b682a02d1e16b69d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "940100f844c94726b682a02d1e16b69d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:00.299151 8032 tablet_bootstrap.cc:654] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:00.305378 8032 tablet_bootstrap.cc:492] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d: No bootstrap required, opened a new log
I20251028 14:53:00.305822 8032 ts_tablet_manager.cc:1403] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20251028 14:53:00.306689 8032 raft_consensus.cc:359] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.306926 8032 raft_consensus.cc:385] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:00.307082 8032 raft_consensus.cc:740] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940100f844c94726b682a02d1e16b69d, State: Initialized, Role: FOLLOWER
I20251028 14:53:00.307413 8032 consensus_queue.cc:260] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.307637 8032 raft_consensus.cc:399] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:00.307763 8032 raft_consensus.cc:493] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:00.307896 8032 raft_consensus.cc:3060] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:00.312511 8032 raft_consensus.cc:515] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.313357 8032 leader_election.cc:304] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 940100f844c94726b682a02d1e16b69d; no voters:
I20251028 14:53:00.313812 8032 leader_election.cc:290] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:00.313922 8034 raft_consensus.cc:2804] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:00.314245 8034 raft_consensus.cc:697] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 1 LEADER]: Becoming Leader. State: Replica: 940100f844c94726b682a02d1e16b69d, State: Running, Role: LEADER
I20251028 14:53:00.314512 8032 ts_tablet_manager.cc:1434] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20251028 14:53:00.314568 8034 consensus_queue.cc:237] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:00.317176 7870 catalog_manager.cc:5649] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d reported cstate change: term changed from 0 to 1, leader changed from <none> to 940100f844c94726b682a02d1e16b69d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "940100f844c94726b682a02d1e16b69d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:00.333760 8027 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:01.165518 7870 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44754:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:01.177685 7987 tablet_service.cc:1505] Processing CreateTablet for tablet 27b1f07c99c3412e93a88f81fd8af1cc (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b8074d8be65140a88fa02476bf2e9e4a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:01.178136 7987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b1f07c99c3412e93a88f81fd8af1cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:01.183372 8050 tablet_bootstrap.cc:492] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: Bootstrap starting.
I20251028 14:53:01.185591 8050 tablet_bootstrap.cc:654] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:01.188201 8050 tablet_bootstrap.cc:492] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: No bootstrap required, opened a new log
I20251028 14:53:01.188413 8050 ts_tablet_manager.cc:1403] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:53:01.189126 8050 raft_consensus.cc:359] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:01.189742 8050 raft_consensus.cc:385] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:01.189826 8050 raft_consensus.cc:740] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940100f844c94726b682a02d1e16b69d, State: Initialized, Role: FOLLOWER
I20251028 14:53:01.190032 8050 consensus_queue.cc:260] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:01.190177 8050 raft_consensus.cc:399] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:01.190241 8050 raft_consensus.cc:493] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:01.190363 8050 raft_consensus.cc:3060] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:01.192292 8050 raft_consensus.cc:515] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:01.192529 8050 leader_election.cc:304] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 940100f844c94726b682a02d1e16b69d; no voters:
I20251028 14:53:01.193226 8050 leader_election.cc:290] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:01.193398 8052 raft_consensus.cc:2804] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:01.194228 8052 raft_consensus.cc:697] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 1 LEADER]: Becoming Leader. State: Replica: 940100f844c94726b682a02d1e16b69d, State: Running, Role: LEADER
I20251028 14:53:01.194715 8052 consensus_queue.cc:237] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } }
I20251028 14:53:01.195142 8050 ts_tablet_manager.cc:1434] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20251028 14:53:01.196228 8054 tablet_replica.cc:442] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "940100f844c94726b682a02d1e16b69d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } } }
I20251028 14:53:01.196640 8054 tablet_replica.cc:445] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:01.196673 8053 tablet_replica.cc:442] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: TxnStatusTablet state changed. Reason: New leader 940100f844c94726b682a02d1e16b69d. Latest consensus state: current_term: 1 leader_uuid: "940100f844c94726b682a02d1e16b69d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } } }
I20251028 14:53:01.197162 8057 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:01.197306 8053 tablet_replica.cc:445] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:01.197396 8057 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:53:01.198120 7870 catalog_manager.cc:5649] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d reported cstate change: term changed from 0 to 1, leader changed from <none> to 940100f844c94726b682a02d1e16b69d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "940100f844c94726b682a02d1e16b69d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940100f844c94726b682a02d1e16b69d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40325 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:01.355427 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:01.378433 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:01.379328 416 tablet_replica.cc:333] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d: stopping tablet replica
I20251028 14:53:01.379554 416 raft_consensus.cc:2243] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:01.379778 416 raft_consensus.cc:2272] T 27b1f07c99c3412e93a88f81fd8af1cc P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:01.380770 416 tablet_replica.cc:333] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d: stopping tablet replica
I20251028 14:53:01.380929 416 raft_consensus.cc:2243] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:01.381096 416 raft_consensus.cc:2272] T c7f565faca914c18bd0d987f83edabac P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:01.382337 416 tablet_replica.cc:333] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d: stopping tablet replica
I20251028 14:53:01.382548 416 raft_consensus.cc:2243] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:01.382730 416 raft_consensus.cc:2272] T 8cc26548c17f4e18963244db9442355c P 940100f844c94726b682a02d1e16b69d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:01.390491 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:01.396054 416 master.cc:561] Master@127.0.104.62:39767 shutting down...
I20251028 14:53:01.409883 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:01.410442 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:01.410652 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4f8ec07b2e3b4f43ae1a9f59deebe85c: stopping tablet replica
I20251028 14:53:01.417392 416 master.cc:583] Master@127.0.104.62:39767 shutdown complete.
[ OK ] TxnOpDispatcherITest.NoPendingWriteOps (1441 ms)
[ RUN ] TxnOpDispatcherITest.NonTransactionalWrites
I20251028 14:53:01.432435 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:41599
I20251028 14:53:01.433060 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:01.435928 8071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:01.436300 8072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:01.436970 8074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:01.437143 416 server_base.cc:1047] running on GCE node
I20251028 14:53:01.437829 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:01.437929 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:01.437968 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663181437953 us; error 0 us; skew 500 ppm
I20251028 14:53:01.438138 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:01.439905 416 webserver.cc:492] Webserver started at http://127.0.104.62:45039/ using document root <none> and password file <none>
I20251028 14:53:01.440173 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:01.440248 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:01.440374 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:01.441131 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "8f42e2996348438ba52514f98e049525"
format_stamp: "Formatted at 2025-10-28 14:53:01 on dist-test-slave-0v9m"
I20251028 14:53:01.444352 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251028 14:53:01.447149 8079 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:01.447927 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251028 14:53:01.448148 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "8f42e2996348438ba52514f98e049525"
format_stamp: "Formatted at 2025-10-28 14:53:01 on dist-test-slave-0v9m"
I20251028 14:53:01.448300 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:01.461066 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:01.461844 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:01.482800 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41599
I20251028 14:53:01.482837 8140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41599 every 8 connection(s)
I20251028 14:53:01.485001 8141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:01.491441 8141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525: Bootstrap starting.
I20251028 14:53:01.494567 8141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:01.497076 8141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525: No bootstrap required, opened a new log
I20251028 14:53:01.497851 8141 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f42e2996348438ba52514f98e049525" member_type: VOTER }
I20251028 14:53:01.498019 8141 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:01.498080 8141 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f42e2996348438ba52514f98e049525, State: Initialized, Role: FOLLOWER
I20251028 14:53:01.498279 8141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f42e2996348438ba52514f98e049525" member_type: VOTER }
I20251028 14:53:01.498440 8141 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:01.498490 8141 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:01.498564 8141 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:01.500465 8141 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f42e2996348438ba52514f98e049525" member_type: VOTER }
I20251028 14:53:01.500700 8141 leader_election.cc:304] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f42e2996348438ba52514f98e049525; no voters:
I20251028 14:53:01.501128 8141 leader_election.cc:290] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:01.501451 8144 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:01.501936 8144 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 1 LEADER]: Becoming Leader. State: Replica: 8f42e2996348438ba52514f98e049525, State: Running, Role: LEADER
I20251028 14:53:01.502199 8141 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:01.502386 8144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f42e2996348438ba52514f98e049525" member_type: VOTER }
I20251028 14:53:01.503325 8145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f42e2996348438ba52514f98e049525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f42e2996348438ba52514f98e049525" member_type: VOTER } }
I20251028 14:53:01.503669 8145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:01.503365 8146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f42e2996348438ba52514f98e049525. Latest consensus state: current_term: 1 leader_uuid: "8f42e2996348438ba52514f98e049525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f42e2996348438ba52514f98e049525" member_type: VOTER } }
I20251028 14:53:01.504711 8146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:01.506070 8150 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:01.507714 8150 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:01.509189 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:01.512102 8150 catalog_manager.cc:1357] Generated new cluster ID: bfedb76f182b4c8291baeae589729887
I20251028 14:53:01.512250 8150 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:01.522800 8150 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:01.523797 8150 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:01.539163 8150 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525: Generated new TSK 0
I20251028 14:53:01.539543 8150 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:01.543399 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:01.547212 8169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:01.547309 8168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:01.550151 416 server_base.cc:1047] running on GCE node
W20251028 14:53:01.550099 8171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:01.550820 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:01.550920 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:01.551002 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663181550966 us; error 0 us; skew 500 ppm
I20251028 14:53:01.551246 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:01.551900 8096 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41118:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:01.553216 416 webserver.cc:492] Webserver started at http://127.0.104.1:33229/ using document root <none> and password file <none>
I20251028 14:53:01.553764 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:01.553992 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:01.554162 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:01.554910 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "5e67f5df0c9f4cc5b802837ac66a0a39"
format_stamp: "Formatted at 2025-10-28 14:53:01 on dist-test-slave-0v9m"
I20251028 14:53:01.557947 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20251028 14:53:01.560226 8176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:01.560838 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251028 14:53:01.560982 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "5e67f5df0c9f4cc5b802837ac66a0a39"
format_stamp: "Formatted at 2025-10-28 14:53:01 on dist-test-slave-0v9m"
I20251028 14:53:01.561156 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:01.582062 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:01.582600 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:01.585337 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:01.585630 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:01.585788 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:01.585832 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:01.609444 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34253
I20251028 14:53:01.609894 8245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34253 every 8 connection(s)
I20251028 14:53:01.614566 8246 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41599
I20251028 14:53:01.614780 8246 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:01.615185 8246 heartbeater.cc:507] Master 127.0.104.62:41599 requested a full tablet report, sending...
I20251028 14:53:01.616612 8096 ts_manager.cc:194] Registered new tserver with Master: 5e67f5df0c9f4cc5b802837ac66a0a39 (127.0.104.1:34253)
I20251028 14:53:01.617223 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00477547s
I20251028 14:53:01.617877 8096 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41124
I20251028 14:53:01.635457 8096 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41146:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:01.655179 8210 tablet_service.cc:1505] Processing CreateTablet for tablet 7875194735e84be9afba86052768fbaf (DEFAULT_TABLE table=txn_op_dispatcher_test [id=edb6c6428b7a4a43bd287d911bab5619]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:01.655179 8209 tablet_service.cc:1505] Processing CreateTablet for tablet 2ab017df94684156ae556dbfb1356953 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=edb6c6428b7a4a43bd287d911bab5619]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:01.655830 8209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ab017df94684156ae556dbfb1356953. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:01.656049 8210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7875194735e84be9afba86052768fbaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:01.664511 8256 tablet_bootstrap.cc:492] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39: Bootstrap starting.
I20251028 14:53:01.666002 8256 tablet_bootstrap.cc:654] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:01.668130 8256 tablet_bootstrap.cc:492] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39: No bootstrap required, opened a new log
I20251028 14:53:01.668322 8256 ts_tablet_manager.cc:1403] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:01.669406 8256 raft_consensus.cc:359] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.669634 8256 raft_consensus.cc:385] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:01.669673 8256 raft_consensus.cc:740] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e67f5df0c9f4cc5b802837ac66a0a39, State: Initialized, Role: FOLLOWER
I20251028 14:53:01.669896 8256 consensus_queue.cc:260] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.670027 8256 raft_consensus.cc:399] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:01.670069 8256 raft_consensus.cc:493] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:01.670148 8256 raft_consensus.cc:3060] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:01.672408 8256 raft_consensus.cc:515] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.672693 8256 leader_election.cc:304] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e67f5df0c9f4cc5b802837ac66a0a39; no voters:
I20251028 14:53:01.673619 8256 leader_election.cc:290] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:01.673782 8258 raft_consensus.cc:2804] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:01.674587 8246 heartbeater.cc:499] Master 127.0.104.62:41599 was elected leader, sending a full tablet report...
I20251028 14:53:01.674638 8256 ts_tablet_manager.cc:1434] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20251028 14:53:01.674822 8258 raft_consensus.cc:697] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 LEADER]: Becoming Leader. State: Replica: 5e67f5df0c9f4cc5b802837ac66a0a39, State: Running, Role: LEADER
I20251028 14:53:01.675096 8256 tablet_bootstrap.cc:492] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39: Bootstrap starting.
I20251028 14:53:01.675087 8258 consensus_queue.cc:237] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.677333 8256 tablet_bootstrap.cc:654] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:01.678293 8096 catalog_manager.cc:5649] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e67f5df0c9f4cc5b802837ac66a0a39 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:01.679160 8256 tablet_bootstrap.cc:492] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39: No bootstrap required, opened a new log
I20251028 14:53:01.679308 8256 ts_tablet_manager.cc:1403] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251028 14:53:01.679939 8256 raft_consensus.cc:359] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.680079 8256 raft_consensus.cc:385] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:01.680128 8256 raft_consensus.cc:740] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e67f5df0c9f4cc5b802837ac66a0a39, State: Initialized, Role: FOLLOWER
I20251028 14:53:01.680328 8256 consensus_queue.cc:260] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.680492 8256 raft_consensus.cc:399] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:01.680588 8256 raft_consensus.cc:493] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:01.680672 8256 raft_consensus.cc:3060] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:01.683423 8256 raft_consensus.cc:515] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.683940 8256 leader_election.cc:304] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e67f5df0c9f4cc5b802837ac66a0a39; no voters:
I20251028 14:53:01.684355 8256 leader_election.cc:290] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:01.684547 8258 raft_consensus.cc:2804] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:01.684931 8258 raft_consensus.cc:697] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 LEADER]: Becoming Leader. State: Replica: 5e67f5df0c9f4cc5b802837ac66a0a39, State: Running, Role: LEADER
I20251028 14:53:01.685086 8256 ts_tablet_manager.cc:1434] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20251028 14:53:01.685449 8258 consensus_queue.cc:237] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } }
I20251028 14:53:01.688818 8096 catalog_manager.cc:5649] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e67f5df0c9f4cc5b802837ac66a0a39 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e67f5df0c9f4cc5b802837ac66a0a39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34253 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:01.717223 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:01.729526 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:01.730381 416 tablet_replica.cc:333] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39: stopping tablet replica
I20251028 14:53:01.730721 416 raft_consensus.cc:2243] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:01.731004 416 raft_consensus.cc:2272] T 2ab017df94684156ae556dbfb1356953 P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:01.732133 416 tablet_replica.cc:333] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39: stopping tablet replica
I20251028 14:53:01.732359 416 raft_consensus.cc:2243] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:01.732668 416 raft_consensus.cc:2272] T 7875194735e84be9afba86052768fbaf P 5e67f5df0c9f4cc5b802837ac66a0a39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:01.750175 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:01.755756 416 master.cc:561] Master@127.0.104.62:41599 shutting down...
I20251028 14:53:02.556490 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:02.556849 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:02.557013 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8f42e2996348438ba52514f98e049525: stopping tablet replica
I20251028 14:53:02.572094 416 master.cc:583] Master@127.0.104.62:41599 shutdown complete.
[ OK ] TxnOpDispatcherITest.NonTransactionalWrites (1151 ms)
[ RUN ] TxnOpDispatcherITest.PreliminaryTasksTimeout
I20251028 14:53:02.583439 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:43677
I20251028 14:53:02.584029 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:02.587584 8268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:02.587626 8267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:02.588652 416 server_base.cc:1047] running on GCE node
W20251028 14:53:02.588843 8270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:02.589643 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:02.589906 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:02.590170 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663182590146 us; error 0 us; skew 500 ppm
I20251028 14:53:02.590404 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:02.592173 416 webserver.cc:492] Webserver started at http://127.0.104.62:37205/ using document root <none> and password file <none>
I20251028 14:53:02.592521 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:02.592622 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:02.592795 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:02.594154 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "764963fb1b5b4f46b614000a872230fb"
format_stamp: "Formatted at 2025-10-28 14:53:02 on dist-test-slave-0v9m"
I20251028 14:53:02.597656 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:53:02.600011 8275 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:02.600499 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251028 14:53:02.600646 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "764963fb1b5b4f46b614000a872230fb"
format_stamp: "Formatted at 2025-10-28 14:53:02 on dist-test-slave-0v9m"
I20251028 14:53:02.600793 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:02.611011 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:02.611661 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:02.630542 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43677
I20251028 14:53:02.630560 8336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43677 every 8 connection(s)
I20251028 14:53:02.632192 8337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:02.640403 8337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb: Bootstrap starting.
I20251028 14:53:02.642859 8337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:02.646018 8337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb: No bootstrap required, opened a new log
I20251028 14:53:02.647190 8337 raft_consensus.cc:359] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764963fb1b5b4f46b614000a872230fb" member_type: VOTER }
I20251028 14:53:02.647403 8337 raft_consensus.cc:385] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:02.647516 8337 raft_consensus.cc:740] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764963fb1b5b4f46b614000a872230fb, State: Initialized, Role: FOLLOWER
I20251028 14:53:02.647763 8337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764963fb1b5b4f46b614000a872230fb" member_type: VOTER }
I20251028 14:53:02.647938 8337 raft_consensus.cc:399] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:02.648172 8337 raft_consensus.cc:493] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:02.648289 8337 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:02.651357 8337 raft_consensus.cc:515] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764963fb1b5b4f46b614000a872230fb" member_type: VOTER }
I20251028 14:53:02.651772 8337 leader_election.cc:304] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 764963fb1b5b4f46b614000a872230fb; no voters:
I20251028 14:53:02.652658 8337 leader_election.cc:290] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:02.652779 8340 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:02.653455 8340 raft_consensus.cc:697] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 1 LEADER]: Becoming Leader. State: Replica: 764963fb1b5b4f46b614000a872230fb, State: Running, Role: LEADER
I20251028 14:53:02.653802 8340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764963fb1b5b4f46b614000a872230fb" member_type: VOTER }
I20251028 14:53:02.654045 8337 sys_catalog.cc:565] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:02.654898 8342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764963fb1b5b4f46b614000a872230fb. Latest consensus state: current_term: 1 leader_uuid: "764963fb1b5b4f46b614000a872230fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764963fb1b5b4f46b614000a872230fb" member_type: VOTER } }
I20251028 14:53:02.654983 8341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "764963fb1b5b4f46b614000a872230fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764963fb1b5b4f46b614000a872230fb" member_type: VOTER } }
I20251028 14:53:02.655139 8342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:02.655143 8341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:02.655793 8344 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:02.657446 8344 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:02.660877 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:02.661465 8344 catalog_manager.cc:1357] Generated new cluster ID: 1a33e6e58efe4ac7b843fe6a5d44b404
I20251028 14:53:02.661597 8344 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:02.695168 8344 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:02.696177 8344 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:02.708273 8344 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb: Generated new TSK 0
I20251028 14:53:02.708629 8344 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:02.722805 8292 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40286:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:02.726905 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:02.730206 8364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:02.730331 8367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:02.730233 8365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:02.730973 416 server_base.cc:1047] running on GCE node
I20251028 14:53:02.731252 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:02.731328 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:02.731431 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663182731392 us; error 0 us; skew 500 ppm
I20251028 14:53:02.731878 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:02.733875 416 webserver.cc:492] Webserver started at http://127.0.104.1:38399/ using document root <none> and password file <none>
I20251028 14:53:02.734261 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:02.734385 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:02.734544 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:02.735271 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "8a9e99c3987d4a09ad67b6032886ff4a"
format_stamp: "Formatted at 2025-10-28 14:53:02 on dist-test-slave-0v9m"
I20251028 14:53:02.738175 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:53:02.740163 8372 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:02.740602 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:53:02.740756 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "8a9e99c3987d4a09ad67b6032886ff4a"
format_stamp: "Formatted at 2025-10-28 14:53:02 on dist-test-slave-0v9m"
I20251028 14:53:02.740916 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:02.752465 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:02.753077 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:02.755434 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:02.755575 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:02.755856 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:02.756074 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:02.787495 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46519
I20251028 14:53:02.787508 8441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46519 every 8 connection(s)
I20251028 14:53:02.790272 8442 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43677
I20251028 14:53:02.790521 8442 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:02.790937 8442 heartbeater.cc:507] Master 127.0.104.62:43677 requested a full tablet report, sending...
I20251028 14:53:02.792420 8292 ts_manager.cc:194] Registered new tserver with Master: 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1:46519)
I20251028 14:53:02.793499 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.005003986s
I20251028 14:53:02.794279 8292 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40292
I20251028 14:53:02.807880 8292 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40304:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:02.825542 8406 tablet_service.cc:1505] Processing CreateTablet for tablet 23e5ad188a16408d8d2896c0c668d175 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=c78c6ac394dd4099a7901a403ca2a76c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:02.825594 8407 tablet_service.cc:1505] Processing CreateTablet for tablet f52297ba5a1a4a07949f099b0f5d0037 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=c78c6ac394dd4099a7901a403ca2a76c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:02.826391 8406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23e5ad188a16408d8d2896c0c668d175. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:02.826705 8407 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f52297ba5a1a4a07949f099b0f5d0037. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:02.836414 8452 tablet_bootstrap.cc:492] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a: Bootstrap starting.
I20251028 14:53:02.838308 8452 tablet_bootstrap.cc:654] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:02.840752 8452 tablet_bootstrap.cc:492] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a: No bootstrap required, opened a new log
I20251028 14:53:02.840912 8452 ts_tablet_manager.cc:1403] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251028 14:53:02.841614 8452 raft_consensus.cc:359] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.841775 8452 raft_consensus.cc:385] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:02.841867 8452 raft_consensus.cc:740] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9e99c3987d4a09ad67b6032886ff4a, State: Initialized, Role: FOLLOWER
I20251028 14:53:02.842100 8452 consensus_queue.cc:260] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.842252 8452 raft_consensus.cc:399] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:02.842334 8452 raft_consensus.cc:493] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:02.842468 8452 raft_consensus.cc:3060] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:02.844722 8452 raft_consensus.cc:515] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.845017 8452 leader_election.cc:304] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9e99c3987d4a09ad67b6032886ff4a; no voters:
I20251028 14:53:02.845631 8452 leader_election.cc:290] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:02.845844 8454 raft_consensus.cc:2804] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:02.846343 8442 heartbeater.cc:499] Master 127.0.104.62:43677 was elected leader, sending a full tablet report...
I20251028 14:53:02.846351 8454 raft_consensus.cc:697] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 LEADER]: Becoming Leader. State: Replica: 8a9e99c3987d4a09ad67b6032886ff4a, State: Running, Role: LEADER
I20251028 14:53:02.846410 8452 ts_tablet_manager.cc:1434] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251028 14:53:02.846762 8452 tablet_bootstrap.cc:492] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a: Bootstrap starting.
I20251028 14:53:02.846755 8454 consensus_queue.cc:237] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.848639 8452 tablet_bootstrap.cc:654] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:02.849442 8292 catalog_manager.cc:5649] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:02.851331 8452 tablet_bootstrap.cc:492] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a: No bootstrap required, opened a new log
I20251028 14:53:02.851495 8452 ts_tablet_manager.cc:1403] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:53:02.852191 8452 raft_consensus.cc:359] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.852890 8452 raft_consensus.cc:385] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:02.853087 8452 raft_consensus.cc:740] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9e99c3987d4a09ad67b6032886ff4a, State: Initialized, Role: FOLLOWER
I20251028 14:53:02.853431 8452 consensus_queue.cc:260] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.853605 8452 raft_consensus.cc:399] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:02.853719 8452 raft_consensus.cc:493] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:02.853856 8452 raft_consensus.cc:3060] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:02.856408 8452 raft_consensus.cc:515] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.856666 8452 leader_election.cc:304] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9e99c3987d4a09ad67b6032886ff4a; no voters:
I20251028 14:53:02.856889 8452 leader_election.cc:290] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:02.857002 8454 raft_consensus.cc:2804] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:02.857226 8454 raft_consensus.cc:697] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 LEADER]: Becoming Leader. State: Replica: 8a9e99c3987d4a09ad67b6032886ff4a, State: Running, Role: LEADER
I20251028 14:53:02.857348 8452 ts_tablet_manager.cc:1434] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251028 14:53:02.857538 8454 consensus_queue.cc:237] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:02.860486 8291 catalog_manager.cc:5649] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:02.879793 8448 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:03.742882 8291 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40320:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:03.754992 8406 tablet_service.cc:1505] Processing CreateTablet for tablet 6bbae1a4caae4382b41df8e70a5a36a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7b10c5f89a184d599d785dfb7f0088f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:03.755630 8406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bbae1a4caae4382b41df8e70a5a36a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:03.761936 8470 tablet_bootstrap.cc:492] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: Bootstrap starting.
I20251028 14:53:03.763660 8470 tablet_bootstrap.cc:654] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:03.766554 8470 tablet_bootstrap.cc:492] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: No bootstrap required, opened a new log
I20251028 14:53:03.766737 8470 ts_tablet_manager.cc:1403] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251028 14:53:03.768283 8470 raft_consensus.cc:359] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:03.768985 8470 raft_consensus.cc:385] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:03.769289 8470 raft_consensus.cc:740] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9e99c3987d4a09ad67b6032886ff4a, State: Initialized, Role: FOLLOWER
I20251028 14:53:03.769589 8470 consensus_queue.cc:260] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:03.769951 8470 raft_consensus.cc:399] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:03.770242 8470 raft_consensus.cc:493] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:03.770710 8470 raft_consensus.cc:3060] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:03.774502 8470 raft_consensus.cc:515] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:03.774760 8470 leader_election.cc:304] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a9e99c3987d4a09ad67b6032886ff4a; no voters:
I20251028 14:53:03.775353 8470 leader_election.cc:290] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:03.775563 8472 raft_consensus.cc:2804] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:03.776336 8472 raft_consensus.cc:697] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 LEADER]: Becoming Leader. State: Replica: 8a9e99c3987d4a09ad67b6032886ff4a, State: Running, Role: LEADER
I20251028 14:53:03.776696 8470 ts_tablet_manager.cc:1434] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20251028 14:53:03.776665 8472 consensus_queue.cc:237] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } }
I20251028 14:53:03.777932 8473 tablet_replica.cc:442] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } } }
I20251028 14:53:03.778108 8473 tablet_replica.cc:445] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:03.779728 8291 catalog_manager.cc:5649] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:03.780030 8474 tablet_replica.cc:442] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: TxnStatusTablet state changed. Reason: New leader 8a9e99c3987d4a09ad67b6032886ff4a. Latest consensus state: current_term: 1 leader_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a9e99c3987d4a09ad67b6032886ff4a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46519 } } }
I20251028 14:53:03.780150 8474 tablet_replica.cc:445] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:03.780164 8476 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:03.780303 8476 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251028 14:53:04.925547 8484 meta_cache.cc:302] tablet 23e5ad188a16408d8d2896c0c668d175: replica 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1:46519) has failed: Timed out: Write RPC to 127.0.104.1:46519 timed out after 0.995s (SENT)
W20251028 14:53:04.925773 8484 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 23e5ad188a16408d8d2896c0c668d175 after 1 attempt(s): Failed to write to server: 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1:46519): Write RPC to 127.0.104.1:46519 timed out after 0.995s (SENT)
W20251028 14:53:05.939447 8495 tablet_replica.cc:1306] Timed out: time out prior registering tablet 23e5ad188a16408d8d2896c0c668d175 as participant (txn ID 0): cancelling pending write operations [suppressed 1 similar messages]
W20251028 14:53:05.939584 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet f52297ba5a1a4a07949f099b0f5d0037 as participant (txn ID 0)
W20251028 14:53:05.939625 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet 23e5ad188a16408d8d2896c0c668d175 as participant (txn ID 0)
W20251028 14:53:05.939944 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=3, attempt_no=0}) took 2004 ms (client timeout 995 ms). Trace:
W20251028 14:53:05.939978 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=0, attempt_no=0}) took 2004 ms (client timeout 997 ms). Trace:
W20251028 14:53:05.940052 8496 rpcz_store.cc:269] 1028 14:53:03.935184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.935297 (+ 113us) service_pool.cc:225] Handling call
1028 14:53:05.939954 (+2004657us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98560,"thread_start_us":322,"threads_started":1,"txn-participant-registration.queue_time_us":622}
W20251028 14:53:05.940035 8495 rpcz_store.cc:269] 1028 14:53:03.935596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.935651 (+ 55us) service_pool.cc:225] Handling call
1028 14:53:05.939914 (+2004263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1107584,"thread_start_us":623,"threads_started":1,"txn-participant-registration.queue_time_us":518}
W20251028 14:53:05.940389 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet 23e5ad188a16408d8d2896c0c668d175 as participant (txn ID 0)
W20251028 14:53:05.940389 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet f52297ba5a1a4a07949f099b0f5d0037 as participant (txn ID 0)
W20251028 14:53:05.940727 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=4, attempt_no=0}) took 2004 ms (client timeout 995 ms). Trace:
W20251028 14:53:05.940791 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=2, attempt_no=0}) took 2005 ms (client timeout 996 ms). Trace:
W20251028 14:53:05.940799 8495 rpcz_store.cc:269] 1028 14:53:03.935853 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.936288 (+ 435us) service_pool.cc:225] Handling call
1028 14:53:03.939770 (+ 3482us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x60d0002b1890. stack: 00007f1a8f5589f6 00007f1a8f5587b0 00007f1a8e30f3ad 000000000050b4c4 000000000050b48d 00000000004ebd1a 00007f1a9b35dc3c 00007f1a9b35d150 00007f1aa2105d6f 00007f1a9c10c116 00007f1a9c10be45 00007f1a9600627e 00007f1a9600515e 00007f1a9600b395 00007f1a9600bcbf 00007f1a9600bae8
1028 14:53:05.940711 (+2000941us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6448384}
W20251028 14:53:05.940879 8496 rpcz_store.cc:269] 1028 14:53:03.935473 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.935575 (+ 102us) service_pool.cc:225] Handling call
1028 14:53:03.939537 (+ 3962us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x60d00021ce60. stack: 00007f1a8f5589f6 00007f1a8f5587b0 00007f1a8e30f3ad 000000000050b4c4 000000000050b48d 00000000004ebd1a 00007f1a9b35dc3c 00007f1a9b35d150 00007f1aa2105d6f 00007f1a9c10c116 00007f1a9c10be45 00007f1a9600627e 00007f1a9600515e 00007f1a9600b395 00007f1a9600bcbf 00007f1a9600bae8
1028 14:53:05.940754 (+2001217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7066880}
W20251028 14:53:05.941067 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet f52297ba5a1a4a07949f099b0f5d0037 as participant (txn ID 0)
W20251028 14:53:05.941076 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet 23e5ad188a16408d8d2896c0c668d175 as participant (txn ID 0)
W20251028 14:53:05.941282 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=1, attempt_no=0}) took 2005 ms (client timeout 996 ms). Trace:
W20251028 14:53:05.941308 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=5, attempt_no=0}) took 2005 ms (client timeout 995 ms). Trace:
W20251028 14:53:05.941361 8496 rpcz_store.cc:269] 1028 14:53:03.935320 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.935416 (+ 96us) service_pool.cc:225] Handling call
1028 14:53:03.939682 (+ 4266us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x60d00021ce60. stack: 00007f1a8f5589f6 00007f1a8f5587b0 00007f1a8e30f3ad 000000000050b4c4 000000000050b48d 00000000004ebd1a 00007f1a9b35dc3c 00007f1a9b35d150 00007f1aa2105d6f 00007f1a9c10c116 00007f1a9c10be45 00007f1a9600627e 00007f1a9600515e 00007f1a9600b395 00007f1a9600bcbf 00007f1a9600bae8
1028 14:53:05.941233 (+2001551us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7881344}
W20251028 14:53:05.941376 8495 rpcz_store.cc:269] 1028 14:53:03.935996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.936638 (+ 642us) service_pool.cc:225] Handling call
1028 14:53:03.939700 (+ 3062us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x60d0002b1890. stack: 00007f1a8f5589f6 00007f1a8f5587b0 00007f1a8e30f3ad 000000000050b4c4 000000000050b48d 00000000004ebd1a 00007f1a9b35dc3c 00007f1a9b35d150 00007f1aa2105d6f 00007f1a9c10c116 00007f1a9c10be45 00007f1a9600627e 00007f1a9600515e 00007f1a9600b395 00007f1a9600bcbf 00007f1a9600bae8
1028 14:53:05.941289 (+2001589us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5771520}
W20251028 14:53:05.941536 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet 23e5ad188a16408d8d2896c0c668d175 as participant (txn ID 0)
W20251028 14:53:05.941686 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=7, attempt_no=0}) took 2005 ms (client timeout 994 ms). Trace:
W20251028 14:53:05.941757 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:58690: Timed out: time out prior registering tablet f52297ba5a1a4a07949f099b0f5d0037 as participant (txn ID 0)
W20251028 14:53:05.941745 8495 rpcz_store.cc:269] 1028 14:53:03.936231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.936589 (+ 358us) service_pool.cc:225] Handling call
1028 14:53:03.939533 (+ 2944us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x60d0002b1890. stack: 00007f1a8f5589f6 00007f1a8f5587b0 00007f1a8e30f3ad 000000000050b4c4 000000000050b48d 00000000004ebd1a 00007f1a9b35dc3c 00007f1a9b35d150 00007f1aa2105d6f 00007f1a9c10c116 00007f1a9c10be45 00007f1a9600627e 00007f1a9600515e 00007f1a9600b395 00007f1a9600bcbf 00007f1a9600bae8
1028 14:53:05.941672 (+2002139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5761280}
W20251028 14:53:05.941962 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58690 (ReqId={client: aec0aed69fef411e949eaa2a34ea28ac, seq_no=6, attempt_no=0}) took 2005 ms (client timeout 995 ms). Trace:
W20251028 14:53:05.942036 8496 rpcz_store.cc:269] 1028 14:53:03.936114 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:03.936192 (+ 78us) service_pool.cc:225] Handling call
1028 14:53:03.939625 (+ 3433us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x60d00021ce60. stack: 00007f1a8f5589f6 00007f1a8f5587b0 00007f1a8e30f3ad 000000000050b4c4 000000000050b48d 00000000004ebd1a 00007f1a9b35dc3c 00007f1a9b35d150 00007f1aa2105d6f 00007f1a9c10c116 00007f1a9c10be45 00007f1a9600627e 00007f1a9600515e 00007f1a9600b395 00007f1a9600bcbf 00007f1a9600bae8
1028 14:53:05.941947 (+2002322us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6469504}
W20251028 14:53:07.001430 8508 meta_cache.cc:302] tablet 23e5ad188a16408d8d2896c0c668d175: replica 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1:46519) has failed: Timed out: Write RPC to 127.0.104.1:46519 timed out after 0.996s (SENT)
W20251028 14:53:07.001607 8508 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 23e5ad188a16408d8d2896c0c668d175 after 1 attempt(s): Failed to write to server: 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1:46519): Write RPC to 127.0.104.1:46519 timed out after 0.996s (SENT)
W20251028 14:53:08.020852 8496 tablet_replica.cc:1306] Timed out: time out prior submitting BEGIN_TXN for participant f52297ba5a1a4a07949f099b0f5d0037 (txn ID 1): cancelling pending write operations [suppressed 1 similar messages]
W20251028 14:53:08.021013 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant f52297ba5a1a4a07949f099b0f5d0037 (txn ID 1)
W20251028 14:53:08.021055 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant 23e5ad188a16408d8d2896c0c668d175 (txn ID 1)
W20251028 14:53:08.021342 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=0, attempt_no=0}) took 2011 ms (client timeout 998 ms). Trace:
W20251028 14:53:08.021416 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=1, attempt_no=0}) took 2011 ms (client timeout 997 ms). Trace:
W20251028 14:53:08.021490 8495 rpcz_store.cc:269] 1028 14:53:06.009702 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.009801 (+ 99us) service_pool.cc:225] Handling call
1028 14:53:08.021312 (+2011511us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3456,"txn-participant-registration.queue_time_us":71}
W20251028 14:53:08.021438 8496 rpcz_store.cc:269] 1028 14:53:06.009547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.009629 (+ 82us) service_pool.cc:225] Handling call
1028 14:53:08.021312 (+2011683us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97536,"txn-participant-registration.queue_time_us":88}
W20251028 14:53:08.021780 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant 23e5ad188a16408d8d2896c0c668d175 (txn ID 1)
W20251028 14:53:08.021780 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant f52297ba5a1a4a07949f099b0f5d0037 (txn ID 1)
W20251028 14:53:08.021981 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=5, attempt_no=0}) took 2011 ms (client timeout 996 ms). Trace:
W20251028 14:53:08.021982 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=2, attempt_no=0}) took 2012 ms (client timeout 996 ms). Trace:
W20251028 14:53:08.022085 8495 rpcz_store.cc:269] 1028 14:53:06.010080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.010285 (+ 205us) service_pool.cc:225] Handling call
1028 14:53:08.021964 (+2011679us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251028 14:53:08.022125 8496 rpcz_store.cc:269] 1028 14:53:06.009808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.009891 (+ 83us) service_pool.cc:225] Handling call
1028 14:53:08.021970 (+2012079us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3968}
W20251028 14:53:08.022259 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant 23e5ad188a16408d8d2896c0c668d175 (txn ID 1)
W20251028 14:53:08.022259 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant f52297ba5a1a4a07949f099b0f5d0037 (txn ID 1)
W20251028 14:53:08.022428 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=3, attempt_no=0}) took 2012 ms (client timeout 996 ms). Trace:
W20251028 14:53:08.022419 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=7, attempt_no=0}) took 2012 ms (client timeout 995 ms). Trace:
W20251028 14:53:08.022504 8496 rpcz_store.cc:269] 1028 14:53:06.009907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.010106 (+ 199us) service_pool.cc:225] Handling call
1028 14:53:08.022414 (+2012308us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251028 14:53:08.022547 8495 rpcz_store.cc:269] 1028 14:53:06.010240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.010495 (+ 255us) service_pool.cc:225] Handling call
1028 14:53:08.022403 (+2011908us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251028 14:53:08.022632 8496 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant f52297ba5a1a4a07949f099b0f5d0037 (txn ID 1)
W20251028 14:53:08.022810 8496 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=4, attempt_no=0}) took 2012 ms (client timeout 996 ms). Trace:
W20251028 14:53:08.022643 8495 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47688: Timed out: time out prior submitting BEGIN_TXN for participant 23e5ad188a16408d8d2896c0c668d175 (txn ID 1)
W20251028 14:53:08.022907 8496 rpcz_store.cc:269] 1028 14:53:06.010000 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.010328 (+ 328us) service_pool.cc:225] Handling call
1028 14:53:08.022794 (+2012466us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251028 14:53:08.023175 8495 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47688 (ReqId={client: a2d6920391924e4498e00b6fe5d8a178, seq_no=6, attempt_no=0}) took 2013 ms (client timeout 995 ms). Trace:
W20251028 14:53:08.023368 8495 rpcz_store.cc:269] 1028 14:53:06.010162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:06.010560 (+ 398us) service_pool.cc:225] Handling call
1028 14:53:08.023150 (+2012590us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251028 14:53:08.040028 8523 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47700: Illegal state: Transaction hasn't been successfully started
W20251028 14:53:08.040524 8527 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47700: Illegal state: Transaction hasn't been successfully started
I20251028 14:53:08.040640 8393 txn_status_manager.cc:244] Participant f52297ba5a1a4a07949f099b0f5d0037 of txn 1 returned error for BEGIN_COMMIT op, aborting: Illegal state: Failed to write to server: 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1:46519): Transaction hasn't been successfully started
I20251028 14:53:08.040933 8393 txn_status_manager.cc:244] Participant 23e5ad188a16408d8d2896c0c668d175 of txn 1 returned error for BEGIN_COMMIT op, aborting: Illegal state: Failed to write to server: 8a9e99c3987d4a09ad67b6032886ff4a (127.0.104.1:46519): Transaction hasn't been successfully started
I20251028 14:53:08.040992 8393 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20251028 14:53:08.059592 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:08.071485 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:08.072160 416 tablet_replica.cc:333] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a: stopping tablet replica
I20251028 14:53:08.072322 416 raft_consensus.cc:2243] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:08.072491 416 raft_consensus.cc:2272] T 23e5ad188a16408d8d2896c0c668d175 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:08.073314 416 tablet_replica.cc:333] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a: stopping tablet replica
I20251028 14:53:08.073446 416 raft_consensus.cc:2243] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:08.073632 416 raft_consensus.cc:2272] T 6bbae1a4caae4382b41df8e70a5a36a1 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:08.074303 416 tablet_replica.cc:333] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a: stopping tablet replica
I20251028 14:53:08.074416 416 raft_consensus.cc:2243] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:08.074549 416 raft_consensus.cc:2272] T f52297ba5a1a4a07949f099b0f5d0037 P 8a9e99c3987d4a09ad67b6032886ff4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:08.080803 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:08.085819 416 master.cc:561] Master@127.0.104.62:43677 shutting down...
I20251028 14:53:08.097136 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:08.097483 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:08.097600 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 764963fb1b5b4f46b614000a872230fb: stopping tablet replica
I20251028 14:53:08.111114 416 master.cc:583] Master@127.0.104.62:43677 shutdown complete.
[ OK ] TxnOpDispatcherITest.PreliminaryTasksTimeout (5545 ms)
[ RUN ] TxnOpDispatcherITest.DuplicateTxnParticipantRegistration
I20251028 14:53:08.129664 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:35411
I20251028 14:53:08.130560 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:08.133935 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:08.133961 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:08.133967 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:08.134524 416 server_base.cc:1047] running on GCE node
I20251028 14:53:08.134812 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:08.134872 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:08.134903 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663188134899 us; error 0 us; skew 500 ppm
I20251028 14:53:08.135049 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:08.136457 416 webserver.cc:492] Webserver started at http://127.0.104.62:33523/ using document root <none> and password file <none>
I20251028 14:53:08.136754 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:08.136824 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:08.136950 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:08.137647 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "fc70bba00438458b8009f37687b822f8"
format_stamp: "Formatted at 2025-10-28 14:53:08 on dist-test-slave-0v9m"
I20251028 14:53:08.140530 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:53:08.142390 8541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:08.142885 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251028 14:53:08.143015 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "fc70bba00438458b8009f37687b822f8"
format_stamp: "Formatted at 2025-10-28 14:53:08 on dist-test-slave-0v9m"
I20251028 14:53:08.143270 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:08.167089 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:08.167702 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:08.186739 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35411
I20251028 14:53:08.186739 8602 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35411 every 8 connection(s)
I20251028 14:53:08.188267 8603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:08.192673 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8: Bootstrap starting.
I20251028 14:53:08.193977 8603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:08.195667 8603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8: No bootstrap required, opened a new log
I20251028 14:53:08.196333 8603 raft_consensus.cc:359] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc70bba00438458b8009f37687b822f8" member_type: VOTER }
I20251028 14:53:08.196468 8603 raft_consensus.cc:385] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:08.196517 8603 raft_consensus.cc:740] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc70bba00438458b8009f37687b822f8, State: Initialized, Role: FOLLOWER
I20251028 14:53:08.196730 8603 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc70bba00438458b8009f37687b822f8" member_type: VOTER }
I20251028 14:53:08.196839 8603 raft_consensus.cc:399] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:08.196898 8603 raft_consensus.cc:493] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:08.197074 8603 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:08.199600 8603 raft_consensus.cc:515] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc70bba00438458b8009f37687b822f8" member_type: VOTER }
I20251028 14:53:08.199847 8603 leader_election.cc:304] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc70bba00438458b8009f37687b822f8; no voters:
I20251028 14:53:08.200276 8603 leader_election.cc:290] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:08.200493 8606 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:08.200927 8606 raft_consensus.cc:697] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 1 LEADER]: Becoming Leader. State: Replica: fc70bba00438458b8009f37687b822f8, State: Running, Role: LEADER
I20251028 14:53:08.201110 8603 sys_catalog.cc:565] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:08.201162 8606 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc70bba00438458b8009f37687b822f8" member_type: VOTER }
I20251028 14:53:08.202188 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc70bba00438458b8009f37687b822f8. Latest consensus state: current_term: 1 leader_uuid: "fc70bba00438458b8009f37687b822f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc70bba00438458b8009f37687b822f8" member_type: VOTER } }
I20251028 14:53:08.202376 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:08.202162 8607 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc70bba00438458b8009f37687b822f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc70bba00438458b8009f37687b822f8" member_type: VOTER } }
I20251028 14:53:08.203461 8612 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:08.203552 8607 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:08.204887 8612 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:08.205948 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:08.209161 8612 catalog_manager.cc:1357] Generated new cluster ID: 948c787911094956bc3b0ecb5136c384
I20251028 14:53:08.209383 8612 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:08.219118 8612 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:08.220165 8612 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:08.232910 8612 catalog_manager.cc:6022] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8: Generated new TSK 0
I20251028 14:53:08.233299 8612 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:08.239370 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251028 14:53:08.241516 8558 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:53:08.242808 8630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:08.242836 8631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:08.243669 8633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:08.244894 416 server_base.cc:1047] running on GCE node
I20251028 14:53:08.245175 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:08.245224 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:08.245275 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663188245242 us; error 0 us; skew 500 ppm
I20251028 14:53:08.245427 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:08.247268 416 webserver.cc:492] Webserver started at http://127.0.104.1:36423/ using document root <none> and password file <none>
I20251028 14:53:08.247686 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:08.247825 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:08.247963 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:08.248636 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "e32c7558deb4434da5f101c061f13e3e"
format_stamp: "Formatted at 2025-10-28 14:53:08 on dist-test-slave-0v9m"
I20251028 14:53:08.251503 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:53:08.253682 8638 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:08.254161 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251028 14:53:08.254276 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "e32c7558deb4434da5f101c061f13e3e"
format_stamp: "Formatted at 2025-10-28 14:53:08 on dist-test-slave-0v9m"
I20251028 14:53:08.254383 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:08.278466 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:08.279031 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:08.280719 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:08.280858 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:08.280993 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:08.281051 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:08.300472 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40765
I20251028 14:53:08.300522 8707 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40765 every 8 connection(s)
I20251028 14:53:08.302418 8708 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35411
I20251028 14:53:08.302587 8708 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:08.302918 8708 heartbeater.cc:507] Master 127.0.104.62:35411 requested a full tablet report, sending...
I20251028 14:53:08.303838 8558 ts_manager.cc:194] Registered new tserver with Master: e32c7558deb4434da5f101c061f13e3e (127.0.104.1:40765)
I20251028 14:53:08.304207 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002360162s
I20251028 14:53:08.305095 8558 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36504
I20251028 14:53:08.318076 8558 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36516:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:08.334805 8670 tablet_service.cc:1505] Processing CreateTablet for tablet 4ce660b8e9f243119e41b79a555d57f7 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=f60030644f6a4562a5f83e32c42c8045]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:08.335052 8668 tablet_service.cc:1505] Processing CreateTablet for tablet 84026e81e3e148778224a70cddf02faa (DEFAULT_TABLE table=txn_op_dispatcher_test [id=f60030644f6a4562a5f83e32c42c8045]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:08.335352 8670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ce660b8e9f243119e41b79a555d57f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:08.335700 8668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84026e81e3e148778224a70cddf02faa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:08.341428 8718 tablet_bootstrap.cc:492] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e: Bootstrap starting.
I20251028 14:53:08.342870 8718 tablet_bootstrap.cc:654] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:08.344678 8718 tablet_bootstrap.cc:492] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e: No bootstrap required, opened a new log
I20251028 14:53:08.345116 8718 ts_tablet_manager.cc:1403] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251028 14:53:08.346243 8718 raft_consensus.cc:359] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.346475 8718 raft_consensus.cc:385] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:08.346515 8718 raft_consensus.cc:740] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32c7558deb4434da5f101c061f13e3e, State: Initialized, Role: FOLLOWER
I20251028 14:53:08.346755 8718 consensus_queue.cc:260] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.346899 8718 raft_consensus.cc:399] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:08.346942 8718 raft_consensus.cc:493] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:08.347003 8718 raft_consensus.cc:3060] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:08.349376 8718 raft_consensus.cc:515] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.349649 8718 leader_election.cc:304] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32c7558deb4434da5f101c061f13e3e; no voters:
I20251028 14:53:08.350307 8718 leader_election.cc:290] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:08.350483 8720 raft_consensus.cc:2804] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:08.351060 8708 heartbeater.cc:499] Master 127.0.104.62:35411 was elected leader, sending a full tablet report...
I20251028 14:53:08.351140 8720 raft_consensus.cc:697] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 1 LEADER]: Becoming Leader. State: Replica: e32c7558deb4434da5f101c061f13e3e, State: Running, Role: LEADER
I20251028 14:53:08.351517 8718 ts_tablet_manager.cc:1434] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251028 14:53:08.351449 8720 consensus_queue.cc:237] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.351861 8718 tablet_bootstrap.cc:492] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e: Bootstrap starting.
I20251028 14:53:08.353835 8718 tablet_bootstrap.cc:654] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:08.354308 8558 catalog_manager.cc:5649] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to e32c7558deb4434da5f101c061f13e3e (127.0.104.1). New cstate: current_term: 1 leader_uuid: "e32c7558deb4434da5f101c061f13e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:08.356096 8718 tablet_bootstrap.cc:492] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e: No bootstrap required, opened a new log
I20251028 14:53:08.356292 8718 ts_tablet_manager.cc:1403] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:08.357120 8718 raft_consensus.cc:359] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.357348 8718 raft_consensus.cc:385] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:08.357455 8718 raft_consensus.cc:740] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32c7558deb4434da5f101c061f13e3e, State: Initialized, Role: FOLLOWER
I20251028 14:53:08.357667 8718 consensus_queue.cc:260] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.357800 8718 raft_consensus.cc:399] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:08.357861 8718 raft_consensus.cc:493] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:08.357949 8718 raft_consensus.cc:3060] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:08.360476 8718 raft_consensus.cc:515] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.360713 8718 leader_election.cc:304] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32c7558deb4434da5f101c061f13e3e; no voters:
I20251028 14:53:08.361126 8718 leader_election.cc:290] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:08.361179 8720 raft_consensus.cc:2804] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:08.361395 8720 raft_consensus.cc:697] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 1 LEADER]: Becoming Leader. State: Replica: e32c7558deb4434da5f101c061f13e3e, State: Running, Role: LEADER
I20251028 14:53:08.361640 8720 consensus_queue.cc:237] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:08.362067 8718 ts_tablet_manager.cc:1434] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20251028 14:53:08.367770 8558 catalog_manager.cc:5649] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to e32c7558deb4434da5f101c061f13e3e (127.0.104.1). New cstate: current_term: 1 leader_uuid: "e32c7558deb4434da5f101c061f13e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:08.378917 8713 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:09.256911 8558 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:09.266045 8670 tablet_service.cc:1505] Processing CreateTablet for tablet f1fc3c0d09044a9ea821691c64919d21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8139b8753d92466b8aadc1573ee86aaf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:09.266621 8670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1fc3c0d09044a9ea821691c64919d21. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:09.272161 8736 tablet_bootstrap.cc:492] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: Bootstrap starting.
I20251028 14:53:09.274098 8736 tablet_bootstrap.cc:654] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:09.275923 8736 tablet_bootstrap.cc:492] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: No bootstrap required, opened a new log
I20251028 14:53:09.276093 8736 ts_tablet_manager.cc:1403] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:09.276765 8736 raft_consensus.cc:359] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:09.276944 8736 raft_consensus.cc:385] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:09.276996 8736 raft_consensus.cc:740] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32c7558deb4434da5f101c061f13e3e, State: Initialized, Role: FOLLOWER
I20251028 14:53:09.277324 8736 consensus_queue.cc:260] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:09.277506 8736 raft_consensus.cc:399] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:09.277575 8736 raft_consensus.cc:493] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:09.277633 8736 raft_consensus.cc:3060] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:09.279531 8736 raft_consensus.cc:515] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:09.279747 8736 leader_election.cc:304] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32c7558deb4434da5f101c061f13e3e; no voters:
I20251028 14:53:09.280185 8736 leader_election.cc:290] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:09.280346 8738 raft_consensus.cc:2804] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:09.280761 8738 raft_consensus.cc:697] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 1 LEADER]: Becoming Leader. State: Replica: e32c7558deb4434da5f101c061f13e3e, State: Running, Role: LEADER
I20251028 14:53:09.280867 8736 ts_tablet_manager.cc:1434] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:53:09.281337 8738 consensus_queue.cc:237] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } }
I20251028 14:53:09.283007 8739 tablet_replica.cc:442] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e32c7558deb4434da5f101c061f13e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } } }
I20251028 14:53:09.282974 8740 tablet_replica.cc:442] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: TxnStatusTablet state changed. Reason: New leader e32c7558deb4434da5f101c061f13e3e. Latest consensus state: current_term: 1 leader_uuid: "e32c7558deb4434da5f101c061f13e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } } }
I20251028 14:53:09.283161 8739 tablet_replica.cc:445] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:09.283180 8740 tablet_replica.cc:445] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:09.283819 8558 catalog_manager.cc:5649] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to e32c7558deb4434da5f101c061f13e3e (127.0.104.1). New cstate: current_term: 1 leader_uuid: "e32c7558deb4434da5f101c061f13e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c7558deb4434da5f101c061f13e3e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40765 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:09.284219 8742 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:09.284379 8742 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251028 14:53:11.207073 8752 meta_cache.cc:302] tablet 4ce660b8e9f243119e41b79a555d57f7: replica e32c7558deb4434da5f101c061f13e3e (127.0.104.1:40765) has failed: Timed out: Write RPC to 127.0.104.1:40765 timed out after 0.998s (SENT)
W20251028 14:53:11.207252 8752 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 4ce660b8e9f243119e41b79a555d57f7 after 1 attempt(s): Failed to write to server: e32c7558deb4434da5f101c061f13e3e (127.0.104.1:40765): Write RPC to 127.0.104.1:40765 timed out after 0.998s (SENT)
W20251028 14:53:12.228047 8757 tablet_replica.cc:1306] Timed out: time out prior submitting BEGIN_TXN for participant 4ce660b8e9f243119e41b79a555d57f7 (txn ID 0): cancelling pending write operations [suppressed 1 similar messages]
W20251028 14:53:12.228236 8757 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:45740: Timed out: time out prior submitting BEGIN_TXN for participant 4ce660b8e9f243119e41b79a555d57f7 (txn ID 0)
W20251028 14:53:12.228520 8757 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:45740 (ReqId={client: b159accdb94d404db9791dfd6dd50c47, seq_no=0, attempt_no=0}) took 2016 ms (client timeout 998 ms). Trace:
W20251028 14:53:12.228617 8757 rpcz_store.cc:269] 1028 14:53:10.212492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:10.212652 (+ 160us) service_pool.cc:225] Handling call
1028 14:53:12.228471 (+2015819us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":316,"threads_started":1,"txn-participant-registration.queue_time_us":436}
I20251028 14:53:12.336416 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:12.358639 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:12.359284 416 tablet_replica.cc:333] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e: stopping tablet replica
I20251028 14:53:12.359553 416 raft_consensus.cc:2243] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:12.359854 416 raft_consensus.cc:2272] T f1fc3c0d09044a9ea821691c64919d21 P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:12.360924 416 tablet_replica.cc:333] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e: stopping tablet replica
I20251028 14:53:12.361136 416 raft_consensus.cc:2243] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:12.361537 416 raft_consensus.cc:2272] T 4ce660b8e9f243119e41b79a555d57f7 P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:12.362519 416 tablet_replica.cc:333] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e: stopping tablet replica
I20251028 14:53:12.362680 416 raft_consensus.cc:2243] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:12.362870 416 raft_consensus.cc:2272] T 84026e81e3e148778224a70cddf02faa P e32c7558deb4434da5f101c061f13e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:12.381858 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:12.392959 416 master.cc:561] Master@127.0.104.62:35411 shutting down...
I20251028 14:53:12.404211 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:12.404516 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:12.404628 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P fc70bba00438458b8009f37687b822f8: stopping tablet replica
I20251028 14:53:12.418668 416 master.cc:583] Master@127.0.104.62:35411 shutdown complete.
[ OK ] TxnOpDispatcherITest.DuplicateTxnParticipantRegistration (4303 ms)
[ RUN ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered
I20251028 14:53:12.433132 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:44493
I20251028 14:53:12.434443 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:12.438390 8773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:12.438760 8774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:12.439543 8776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:12.440487 416 server_base.cc:1047] running on GCE node
I20251028 14:53:12.440757 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:12.440842 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:12.440868 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663192440864 us; error 0 us; skew 500 ppm
I20251028 14:53:12.440992 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:12.442401 416 webserver.cc:492] Webserver started at http://127.0.104.62:43257/ using document root <none> and password file <none>
I20251028 14:53:12.442622 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:12.442678 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:12.442773 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:12.443384 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "ec4b66d06ea04332bd5d240424a49a77"
format_stamp: "Formatted at 2025-10-28 14:53:12 on dist-test-slave-0v9m"
I20251028 14:53:12.446811 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251028 14:53:12.448916 8781 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:12.449618 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251028 14:53:12.449855 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "ec4b66d06ea04332bd5d240424a49a77"
format_stamp: "Formatted at 2025-10-28 14:53:12 on dist-test-slave-0v9m"
I20251028 14:53:12.450021 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:12.473327 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:12.473862 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:12.490064 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44493
I20251028 14:53:12.490093 8842 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44493 every 8 connection(s)
I20251028 14:53:12.491396 8843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:12.496399 8843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77: Bootstrap starting.
I20251028 14:53:12.497690 8843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:12.499853 8843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77: No bootstrap required, opened a new log
I20251028 14:53:12.500571 8843 raft_consensus.cc:359] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4b66d06ea04332bd5d240424a49a77" member_type: VOTER }
I20251028 14:53:12.500721 8843 raft_consensus.cc:385] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:12.500754 8843 raft_consensus.cc:740] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4b66d06ea04332bd5d240424a49a77, State: Initialized, Role: FOLLOWER
I20251028 14:53:12.500967 8843 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4b66d06ea04332bd5d240424a49a77" member_type: VOTER }
I20251028 14:53:12.501076 8843 raft_consensus.cc:399] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:12.501111 8843 raft_consensus.cc:493] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:12.501180 8843 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:12.503155 8843 raft_consensus.cc:515] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4b66d06ea04332bd5d240424a49a77" member_type: VOTER }
I20251028 14:53:12.503364 8843 leader_election.cc:304] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec4b66d06ea04332bd5d240424a49a77; no voters:
I20251028 14:53:12.503800 8843 leader_election.cc:290] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:12.504024 8846 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:12.504463 8846 raft_consensus.cc:697] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 1 LEADER]: Becoming Leader. State: Replica: ec4b66d06ea04332bd5d240424a49a77, State: Running, Role: LEADER
I20251028 14:53:12.504707 8846 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4b66d06ea04332bd5d240424a49a77" member_type: VOTER }
I20251028 14:53:12.504760 8843 sys_catalog.cc:565] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:12.505558 8848 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec4b66d06ea04332bd5d240424a49a77. Latest consensus state: current_term: 1 leader_uuid: "ec4b66d06ea04332bd5d240424a49a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4b66d06ea04332bd5d240424a49a77" member_type: VOTER } }
I20251028 14:53:12.505553 8847 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec4b66d06ea04332bd5d240424a49a77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4b66d06ea04332bd5d240424a49a77" member_type: VOTER } }
I20251028 14:53:12.505786 8847 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:12.506392 8848 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:12.506480 8852 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:12.507758 8852 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:12.511816 8852 catalog_manager.cc:1357] Generated new cluster ID: 9b6f7a87277e4e419d560d5575fdad54
I20251028 14:53:12.511933 8852 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:12.513089 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:12.531425 8852 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:12.532289 8852 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:12.538373 8852 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77: Generated new TSK 0
I20251028 14:53:12.538672 8852 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:12.546962 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251028 14:53:12.548581 8798 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54072:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:53:12.550459 8870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:12.550701 8871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:12.551508 416 server_base.cc:1047] running on GCE node
W20251028 14:53:12.551967 8873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:12.552451 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:12.552546 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:12.552578 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663192552575 us; error 0 us; skew 500 ppm
I20251028 14:53:12.552721 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:12.554551 416 webserver.cc:492] Webserver started at http://127.0.104.1:41109/ using document root <none> and password file <none>
I20251028 14:53:12.554832 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:12.554934 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:12.555065 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:12.555809 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "330bf1c3cc6f4105a975c3cc21b907d6"
format_stamp: "Formatted at 2025-10-28 14:53:12 on dist-test-slave-0v9m"
I20251028 14:53:12.559247 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20251028 14:53:12.561795 8878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:12.562345 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:53:12.562470 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "330bf1c3cc6f4105a975c3cc21b907d6"
format_stamp: "Formatted at 2025-10-28 14:53:12 on dist-test-slave-0v9m"
I20251028 14:53:12.562590 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:12.582871 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:12.583392 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:12.585119 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:12.585197 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:12.585321 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:12.585376 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:12.602461 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40047
I20251028 14:53:12.602490 8947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40047 every 8 connection(s)
I20251028 14:53:12.604140 8948 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44493
I20251028 14:53:12.604256 8948 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:12.604542 8948 heartbeater.cc:507] Master 127.0.104.62:44493 requested a full tablet report, sending...
I20251028 14:53:12.605664 8798 ts_manager.cc:194] Registered new tserver with Master: 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1:40047)
I20251028 14:53:12.605820 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002360709s
I20251028 14:53:12.606763 8798 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54088
I20251028 14:53:12.618943 8798 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54104:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:12.635874 8908 tablet_service.cc:1505] Processing CreateTablet for tablet 5675bd1645b34233a4ed1ceaf003f74c (DEFAULT_TABLE table=txn_op_dispatcher_test [id=702b99116a3b498d8e6d9e9679e5f038]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:12.636008 8907 tablet_service.cc:1505] Processing CreateTablet for tablet a5720b5cdf714825a568ee95658845ef (DEFAULT_TABLE table=txn_op_dispatcher_test [id=702b99116a3b498d8e6d9e9679e5f038]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:12.636442 8908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5675bd1645b34233a4ed1ceaf003f74c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:12.636679 8907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5720b5cdf714825a568ee95658845ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:12.644379 8958 tablet_bootstrap.cc:492] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6: Bootstrap starting.
I20251028 14:53:12.645745 8958 tablet_bootstrap.cc:654] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:12.648118 8958 tablet_bootstrap.cc:492] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6: No bootstrap required, opened a new log
I20251028 14:53:12.648281 8958 ts_tablet_manager.cc:1403] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:12.648906 8958 raft_consensus.cc:359] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.649046 8958 raft_consensus.cc:385] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:12.649093 8958 raft_consensus.cc:740] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bf1c3cc6f4105a975c3cc21b907d6, State: Initialized, Role: FOLLOWER
I20251028 14:53:12.649314 8958 consensus_queue.cc:260] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.649431 8958 raft_consensus.cc:399] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:12.649490 8958 raft_consensus.cc:493] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:12.649560 8958 raft_consensus.cc:3060] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:12.651600 8958 raft_consensus.cc:515] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.651818 8958 leader_election.cc:304] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 330bf1c3cc6f4105a975c3cc21b907d6; no voters:
I20251028 14:53:12.652343 8958 leader_election.cc:290] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:12.652493 8960 raft_consensus.cc:2804] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:12.653179 8948 heartbeater.cc:499] Master 127.0.104.62:44493 was elected leader, sending a full tablet report...
I20251028 14:53:12.653187 8958 ts_tablet_manager.cc:1434] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:53:12.653733 8958 tablet_bootstrap.cc:492] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6: Bootstrap starting.
I20251028 14:53:12.654220 8960 raft_consensus.cc:697] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 LEADER]: Becoming Leader. State: Replica: 330bf1c3cc6f4105a975c3cc21b907d6, State: Running, Role: LEADER
I20251028 14:53:12.654480 8960 consensus_queue.cc:237] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.655437 8958 tablet_bootstrap.cc:654] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:12.657089 8958 tablet_bootstrap.cc:492] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6: No bootstrap required, opened a new log
I20251028 14:53:12.657243 8958 ts_tablet_manager.cc:1403] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:12.657301 8798 catalog_manager.cc:5649] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:12.657992 8958 raft_consensus.cc:359] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.658138 8958 raft_consensus.cc:385] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:12.658188 8958 raft_consensus.cc:740] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bf1c3cc6f4105a975c3cc21b907d6, State: Initialized, Role: FOLLOWER
I20251028 14:53:12.658376 8958 consensus_queue.cc:260] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.658540 8958 raft_consensus.cc:399] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:12.658597 8958 raft_consensus.cc:493] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:12.658680 8958 raft_consensus.cc:3060] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:12.661069 8958 raft_consensus.cc:515] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.661353 8958 leader_election.cc:304] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 330bf1c3cc6f4105a975c3cc21b907d6; no voters:
I20251028 14:53:12.661535 8958 leader_election.cc:290] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:12.661651 8960 raft_consensus.cc:2804] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:12.661790 8960 raft_consensus.cc:697] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 LEADER]: Becoming Leader. State: Replica: 330bf1c3cc6f4105a975c3cc21b907d6, State: Running, Role: LEADER
I20251028 14:53:12.661935 8958 ts_tablet_manager.cc:1434] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:53:12.662002 8960 consensus_queue.cc:237] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:12.664880 8798 catalog_manager.cc:5649] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:12.673238 8954 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:13.572157 8797 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54114:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:13.581457 8908 tablet_service.cc:1505] Processing CreateTablet for tablet e67dd0ffee70465887eab1ab0eefb17b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b255f8845db14d94ac78a387c7b87f72]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:13.581888 8908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67dd0ffee70465887eab1ab0eefb17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:13.587024 8976 tablet_bootstrap.cc:492] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: Bootstrap starting.
I20251028 14:53:13.588721 8976 tablet_bootstrap.cc:654] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:13.591033 8976 tablet_bootstrap.cc:492] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: No bootstrap required, opened a new log
I20251028 14:53:13.591301 8976 ts_tablet_manager.cc:1403] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:13.592278 8976 raft_consensus.cc:359] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:13.592447 8976 raft_consensus.cc:385] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:13.592483 8976 raft_consensus.cc:740] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 330bf1c3cc6f4105a975c3cc21b907d6, State: Initialized, Role: FOLLOWER
I20251028 14:53:13.592679 8976 consensus_queue.cc:260] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:13.592804 8976 raft_consensus.cc:399] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:13.592844 8976 raft_consensus.cc:493] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:13.592903 8976 raft_consensus.cc:3060] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:13.595484 8976 raft_consensus.cc:515] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:13.595763 8976 leader_election.cc:304] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 330bf1c3cc6f4105a975c3cc21b907d6; no voters:
I20251028 14:53:13.596422 8976 leader_election.cc:290] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:13.596578 8978 raft_consensus.cc:2804] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:13.597631 8978 raft_consensus.cc:697] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 LEADER]: Becoming Leader. State: Replica: 330bf1c3cc6f4105a975c3cc21b907d6, State: Running, Role: LEADER
I20251028 14:53:13.597671 8976 ts_tablet_manager.cc:1434] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251028 14:53:13.597937 8978 consensus_queue.cc:237] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } }
I20251028 14:53:13.598973 8979 tablet_replica.cc:442] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } } }
I20251028 14:53:13.599037 8980 tablet_replica.cc:442] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: TxnStatusTablet state changed. Reason: New leader 330bf1c3cc6f4105a975c3cc21b907d6. Latest consensus state: current_term: 1 leader_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } } }
I20251028 14:53:13.599170 8980 tablet_replica.cc:445] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:13.600380 8979 tablet_replica.cc:445] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:13.600603 8982 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:13.600728 8982 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:53:13.600860 8797 catalog_manager.cc:5649] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "330bf1c3cc6f4105a975c3cc21b907d6" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40047 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:15.725128 8990 tablet_replica.cc:1306] Illegal state: Failed to write to server: 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1:40047): transaction ID 0 not open: ABORTED: cancelling pending write operations
W20251028 14:53:15.725405 8990 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:47402: Illegal state: Failed to write to server: 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1:40047): transaction ID 0 not open: ABORTED
I20251028 14:53:15.725828 8990 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:47402 (ReqId={client: be614d80918248ceaf294edccb4b068a, seq_no=0, attempt_no=0}) took 2014 ms. Trace:
I20251028 14:53:15.725942 8990 rpcz_store.cc:276] 1028 14:53:13.711255 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:13.711362 (+ 107us) service_pool.cc:225] Handling call
1028 14:53:15.725702 (+2014340us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":335,"threads_started":1,"txn-participant-registration.queue_time_us":459}
W20251028 14:53:15.726729 8954 batcher.cc:441] Illegal state: Failed to write batch of 1 ops to tablet 5675bd1645b34233a4ed1ceaf003f74c after 1 attempt(s): Failed to write to server: 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1:40047): Failed to write to server: 330bf1c3cc6f4105a975c3cc21b907d6 (127.0.104.1:40047): transaction ID 0 not open: ABORTED
I20251028 14:53:15.742869 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:15.759963 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:15.760525 416 tablet_replica.cc:333] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6: stopping tablet replica
I20251028 14:53:15.760762 416 raft_consensus.cc:2243] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:15.760988 416 raft_consensus.cc:2272] T e67dd0ffee70465887eab1ab0eefb17b P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:15.762243 416 tablet_replica.cc:333] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6: stopping tablet replica
I20251028 14:53:15.762498 416 raft_consensus.cc:2243] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:15.762703 416 raft_consensus.cc:2272] T 5675bd1645b34233a4ed1ceaf003f74c P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:15.763337 416 tablet_replica.cc:333] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6: stopping tablet replica
I20251028 14:53:15.763566 416 raft_consensus.cc:2243] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:15.763963 416 raft_consensus.cc:2272] T a5720b5cdf714825a568ee95658845ef P 330bf1c3cc6f4105a975c3cc21b907d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:15.781525 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:15.787580 416 master.cc:561] Master@127.0.104.62:44493 shutting down...
I20251028 14:53:15.804330 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:15.804631 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:15.804785 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P ec4b66d06ea04332bd5d240424a49a77: stopping tablet replica
I20251028 14:53:15.819977 416 master.cc:583] Master@127.0.104.62:44493 shutdown complete.
[ OK ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered (3400 ms)
[ RUN ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered
I20251028 14:53:15.833155 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:40501
I20251028 14:53:15.833926 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:15.837189 8999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:15.837394 9000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:15.837818 9002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:15.837985 416 server_base.cc:1047] running on GCE node
I20251028 14:53:15.838340 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:15.838398 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:15.838421 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663195838417 us; error 0 us; skew 500 ppm
I20251028 14:53:15.838593 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:15.840401 416 webserver.cc:492] Webserver started at http://127.0.104.62:43227/ using document root <none> and password file <none>
I20251028 14:53:15.840682 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:15.840756 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:15.840878 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:15.841609 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "ca0677e44ead443687c2e600c977fd3e"
format_stamp: "Formatted at 2025-10-28 14:53:15 on dist-test-slave-0v9m"
I20251028 14:53:15.844612 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:53:15.846817 9007 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:15.847402 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251028 14:53:15.847532 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "ca0677e44ead443687c2e600c977fd3e"
format_stamp: "Formatted at 2025-10-28 14:53:15 on dist-test-slave-0v9m"
I20251028 14:53:15.847664 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:15.863341 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:15.864202 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:15.880218 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40501
I20251028 14:53:15.880249 9068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40501 every 8 connection(s)
I20251028 14:53:15.881680 9069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:15.886381 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e: Bootstrap starting.
I20251028 14:53:15.887602 9069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:15.889379 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e: No bootstrap required, opened a new log
I20251028 14:53:15.890056 9069 raft_consensus.cc:359] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0677e44ead443687c2e600c977fd3e" member_type: VOTER }
I20251028 14:53:15.890192 9069 raft_consensus.cc:385] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:15.890226 9069 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca0677e44ead443687c2e600c977fd3e, State: Initialized, Role: FOLLOWER
I20251028 14:53:15.890410 9069 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0677e44ead443687c2e600c977fd3e" member_type: VOTER }
I20251028 14:53:15.890522 9069 raft_consensus.cc:399] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:15.890579 9069 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:15.890652 9069 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:15.892782 9069 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0677e44ead443687c2e600c977fd3e" member_type: VOTER }
I20251028 14:53:15.893029 9069 leader_election.cc:304] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca0677e44ead443687c2e600c977fd3e; no voters:
I20251028 14:53:15.893666 9069 leader_election.cc:290] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:15.894032 9072 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:15.894796 9069 sys_catalog.cc:565] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:15.894855 9072 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 1 LEADER]: Becoming Leader. State: Replica: ca0677e44ead443687c2e600c977fd3e, State: Running, Role: LEADER
I20251028 14:53:15.895128 9072 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0677e44ead443687c2e600c977fd3e" member_type: VOTER }
I20251028 14:53:15.896066 9073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca0677e44ead443687c2e600c977fd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0677e44ead443687c2e600c977fd3e" member_type: VOTER } }
I20251028 14:53:15.896085 9074 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca0677e44ead443687c2e600c977fd3e. Latest consensus state: current_term: 1 leader_uuid: "ca0677e44ead443687c2e600c977fd3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca0677e44ead443687c2e600c977fd3e" member_type: VOTER } }
I20251028 14:53:15.896287 9073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:15.896281 9074 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:15.897468 9077 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:15.898890 9077 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:15.900454 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:15.902145 9077 catalog_manager.cc:1357] Generated new cluster ID: 5c8e283ebe8c466db7851ffdb2b35e3a
I20251028 14:53:15.902235 9077 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:15.928638 9077 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:15.930580 9077 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:15.943665 9077 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e: Generated new TSK 0
I20251028 14:53:15.944178 9077 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:15.952453 9024 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38750:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:15.966460 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:15.969897 9096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:15.969913 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:15.970598 416 server_base.cc:1047] running on GCE node
W20251028 14:53:15.971073 9099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:15.971526 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:15.971611 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:15.971715 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663195971691 us; error 0 us; skew 500 ppm
I20251028 14:53:15.971905 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:15.973531 416 webserver.cc:492] Webserver started at http://127.0.104.1:39861/ using document root <none> and password file <none>
I20251028 14:53:15.973891 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:15.974004 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:15.974169 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:15.975214 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb"
format_stamp: "Formatted at 2025-10-28 14:53:15 on dist-test-slave-0v9m"
I20251028 14:53:15.979507 416 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:15.982686 9104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:15.983144 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:53:15.983294 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb"
format_stamp: "Formatted at 2025-10-28 14:53:15 on dist-test-slave-0v9m"
I20251028 14:53:15.983454 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:16.017056 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:16.017676 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:16.019752 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:16.019846 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:16.019930 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:16.019974 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:16.037828 9173 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37751 every 8 connection(s)
I20251028 14:53:16.037822 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37751
I20251028 14:53:16.040385 9174 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40501
I20251028 14:53:16.040544 9174 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:16.040935 9174 heartbeater.cc:507] Master 127.0.104.62:40501 requested a full tablet report, sending...
I20251028 14:53:16.042073 9024 ts_manager.cc:194] Registered new tserver with Master: 47cd9e0abbb84ad092fa05a6dd2f3bfb (127.0.104.1:37751)
I20251028 14:53:16.043066 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.001226711s
I20251028 14:53:16.043641 9024 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38752
I20251028 14:53:16.057082 9024 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38760:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:16.073761 9134 tablet_service.cc:1505] Processing CreateTablet for tablet a332a7e5f4674214b0c77c790b4f2430 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=03b1a34f3cfc40619860646a453abcc2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:16.073761 9133 tablet_service.cc:1505] Processing CreateTablet for tablet cdc2a6d43d2e4e0aafd377e82b0a4faf (DEFAULT_TABLE table=txn_op_dispatcher_test [id=03b1a34f3cfc40619860646a453abcc2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:16.074690 9134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a332a7e5f4674214b0c77c790b4f2430. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:16.075107 9133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdc2a6d43d2e4e0aafd377e82b0a4faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:16.083452 9184 tablet_bootstrap.cc:492] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Bootstrap starting.
I20251028 14:53:16.085139 9184 tablet_bootstrap.cc:654] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:16.088047 9184 tablet_bootstrap.cc:492] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: No bootstrap required, opened a new log
I20251028 14:53:16.088269 9184 ts_tablet_manager.cc:1403] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:53:16.089365 9184 raft_consensus.cc:359] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.089641 9184 raft_consensus.cc:385] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:16.089709 9184 raft_consensus.cc:740] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cd9e0abbb84ad092fa05a6dd2f3bfb, State: Initialized, Role: FOLLOWER
I20251028 14:53:16.089921 9184 consensus_queue.cc:260] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.090039 9184 raft_consensus.cc:399] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:16.090075 9184 raft_consensus.cc:493] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:16.090130 9184 raft_consensus.cc:3060] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:16.092187 9184 raft_consensus.cc:515] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.092437 9184 leader_election.cc:304] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cd9e0abbb84ad092fa05a6dd2f3bfb; no voters:
I20251028 14:53:16.093016 9184 leader_election.cc:290] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:16.093127 9186 raft_consensus.cc:2804] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:16.093902 9186 raft_consensus.cc:697] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 LEADER]: Becoming Leader. State: Replica: 47cd9e0abbb84ad092fa05a6dd2f3bfb, State: Running, Role: LEADER
I20251028 14:53:16.093921 9174 heartbeater.cc:499] Master 127.0.104.62:40501 was elected leader, sending a full tablet report...
I20251028 14:53:16.093952 9184 ts_tablet_manager.cc:1434] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20251028 14:53:16.094236 9186 consensus_queue.cc:237] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.094561 9184 tablet_bootstrap.cc:492] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Bootstrap starting.
I20251028 14:53:16.096570 9184 tablet_bootstrap.cc:654] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:16.097970 9024 catalog_manager.cc:5649] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 47cd9e0abbb84ad092fa05a6dd2f3bfb (127.0.104.1). New cstate: current_term: 1 leader_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:16.099697 9184 tablet_bootstrap.cc:492] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb: No bootstrap required, opened a new log
I20251028 14:53:16.100014 9184 ts_tablet_manager.cc:1403] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20251028 14:53:16.100981 9184 raft_consensus.cc:359] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.101419 9184 raft_consensus.cc:385] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:16.101574 9184 raft_consensus.cc:740] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cd9e0abbb84ad092fa05a6dd2f3bfb, State: Initialized, Role: FOLLOWER
I20251028 14:53:16.101912 9184 consensus_queue.cc:260] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.102079 9184 raft_consensus.cc:399] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:16.102142 9184 raft_consensus.cc:493] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:16.102219 9184 raft_consensus.cc:3060] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:16.105043 9184 raft_consensus.cc:515] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.105302 9184 leader_election.cc:304] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cd9e0abbb84ad092fa05a6dd2f3bfb; no voters:
I20251028 14:53:16.105482 9184 leader_election.cc:290] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:16.105551 9186 raft_consensus.cc:2804] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:16.105829 9186 raft_consensus.cc:697] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 LEADER]: Becoming Leader. State: Replica: 47cd9e0abbb84ad092fa05a6dd2f3bfb, State: Running, Role: LEADER
I20251028 14:53:16.106142 9186 consensus_queue.cc:237] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.105901 9184 ts_tablet_manager.cc:1434] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20251028 14:53:16.108134 9024 catalog_manager.cc:5649] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 47cd9e0abbb84ad092fa05a6dd2f3bfb (127.0.104.1). New cstate: current_term: 1 leader_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:16.125370 9179 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:16.966871 9024 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38764:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:16.975342 9134 tablet_service.cc:1505] Processing CreateTablet for tablet 4d2f96509de2487ab47e632dbab50ac8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9b8a628a4ecf4b7fa53b6fb63ee70df0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:16.975811 9134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2f96509de2487ab47e632dbab50ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:16.981072 9202 tablet_bootstrap.cc:492] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Bootstrap starting.
I20251028 14:53:16.982638 9202 tablet_bootstrap.cc:654] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:16.984293 9202 tablet_bootstrap.cc:492] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: No bootstrap required, opened a new log
I20251028 14:53:16.984436 9202 ts_tablet_manager.cc:1403] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:16.985116 9202 raft_consensus.cc:359] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.985307 9202 raft_consensus.cc:385] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:16.985371 9202 raft_consensus.cc:740] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cd9e0abbb84ad092fa05a6dd2f3bfb, State: Initialized, Role: FOLLOWER
I20251028 14:53:16.985555 9202 consensus_queue.cc:260] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.985692 9202 raft_consensus.cc:399] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:16.985755 9202 raft_consensus.cc:493] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:16.985816 9202 raft_consensus.cc:3060] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:16.987730 9202 raft_consensus.cc:515] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.987917 9202 leader_election.cc:304] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47cd9e0abbb84ad092fa05a6dd2f3bfb; no voters:
I20251028 14:53:16.988323 9202 leader_election.cc:290] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:16.988474 9204 raft_consensus.cc:2804] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:16.988888 9204 raft_consensus.cc:697] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 LEADER]: Becoming Leader. State: Replica: 47cd9e0abbb84ad092fa05a6dd2f3bfb, State: Running, Role: LEADER
I20251028 14:53:16.988969 9202 ts_tablet_manager.cc:1434] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20251028 14:53:16.989128 9204 consensus_queue.cc:237] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } }
I20251028 14:53:16.990207 9206 tablet_replica.cc:442] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: TxnStatusTablet state changed. Reason: New leader 47cd9e0abbb84ad092fa05a6dd2f3bfb. Latest consensus state: current_term: 1 leader_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } } }
I20251028 14:53:16.990342 9206 tablet_replica.cc:445] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:16.990695 9205 tablet_replica.cc:442] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } } }
I20251028 14:53:16.990823 9205 tablet_replica.cc:445] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:16.991604 9024 catalog_manager.cc:5649] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 47cd9e0abbb84ad092fa05a6dd2f3bfb (127.0.104.1). New cstate: current_term: 1 leader_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47cd9e0abbb84ad092fa05a6dd2f3bfb" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37751 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:16.993516 9208 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:16.993666 9208 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251028 14:53:18.238763 9219 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
W20251028 14:53:18.238899 9219 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:34848: Aborted: operation has been aborted
I20251028 14:53:18.239118 9219 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:34848 (ReqId={client: 933a0fa9c0a04f9da2389c96461d3722, seq_no=0, attempt_no=0}) took 1001 ms. Trace:
I20251028 14:53:18.239185 9219 rpcz_store.cc:276] 1028 14:53:17.238084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:17.238174 (+ 90us) service_pool.cc:225] Handling call
1028 14:53:18.239103 (+1000929us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":329,"threads_started":1,"txn-participant-registration.queue_time_us":450}
W20251028 14:53:18.239804 9179 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet a332a7e5f4674214b0c77c790b4f2430 after 1 attempt(s): Failed to write to server: 47cd9e0abbb84ad092fa05a6dd2f3bfb (127.0.104.1:37751): operation has been aborted
I20251028 14:53:18.268159 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:18.280458 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20251028 14:53:19.261330 9224 tablet_replica.cc:1306] Illegal state: Transaction metadata for transaction 0 already exists: cancelling pending write operations
I20251028 14:53:19.261394 416 tablet_replica.cc:333] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: stopping tablet replica
I20251028 14:53:19.261668 416 raft_consensus.cc:2243] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:19.262094 416 raft_consensus.cc:2272] T 4d2f96509de2487ab47e632dbab50ac8 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:19.263144 416 tablet_replica.cc:333] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb: stopping tablet replica
I20251028 14:53:19.263309 416 raft_consensus.cc:2243] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:19.263481 416 raft_consensus.cc:2272] T a332a7e5f4674214b0c77c790b4f2430 P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:19.264309 416 tablet_replica.cc:333] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb: stopping tablet replica
I20251028 14:53:19.264539 416 raft_consensus.cc:2243] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:19.264771 416 raft_consensus.cc:2272] T cdc2a6d43d2e4e0aafd377e82b0a4faf P 47cd9e0abbb84ad092fa05a6dd2f3bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:19.280709 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:19.287060 416 master.cc:561] Master@127.0.104.62:40501 shutting down...
I20251028 14:53:19.301330 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:19.301669 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:19.301846 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P ca0677e44ead443687c2e600c977fd3e: stopping tablet replica
I20251028 14:53:19.317169 416 master.cc:583] Master@127.0.104.62:40501 shutdown complete.
[ OK ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered (3497 ms)
[ RUN ] TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted
I20251028 14:53:19.329919 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:43241
I20251028 14:53:19.330506 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:19.333700 9225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:19.333700 9226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:19.334306 9228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:19.334510 416 server_base.cc:1047] running on GCE node
I20251028 14:53:19.335034 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:19.335136 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:19.335191 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663199335171 us; error 0 us; skew 500 ppm
I20251028 14:53:19.335376 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:19.337147 416 webserver.cc:492] Webserver started at http://127.0.104.62:41735/ using document root <none> and password file <none>
I20251028 14:53:19.337486 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:19.337574 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:19.337761 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:19.338596 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "8e1b824970ab41a18b25890b19dadba0"
format_stamp: "Formatted at 2025-10-28 14:53:19 on dist-test-slave-0v9m"
I20251028 14:53:19.341507 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20251028 14:53:19.343367 9233 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:19.343736 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251028 14:53:19.343860 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "8e1b824970ab41a18b25890b19dadba0"
format_stamp: "Formatted at 2025-10-28 14:53:19 on dist-test-slave-0v9m"
I20251028 14:53:19.344020 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:19.353973 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:19.354562 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:19.371132 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43241
I20251028 14:53:19.371155 9294 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43241 every 8 connection(s)
I20251028 14:53:19.372500 9295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:19.376921 9295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0: Bootstrap starting.
I20251028 14:53:19.378238 9295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:19.379995 9295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0: No bootstrap required, opened a new log
I20251028 14:53:19.380640 9295 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1b824970ab41a18b25890b19dadba0" member_type: VOTER }
I20251028 14:53:19.380800 9295 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:19.380856 9295 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1b824970ab41a18b25890b19dadba0, State: Initialized, Role: FOLLOWER
I20251028 14:53:19.381060 9295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1b824970ab41a18b25890b19dadba0" member_type: VOTER }
I20251028 14:53:19.381193 9295 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:19.381286 9295 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:19.381385 9295 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:19.383314 9295 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1b824970ab41a18b25890b19dadba0" member_type: VOTER }
I20251028 14:53:19.383531 9295 leader_election.cc:304] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e1b824970ab41a18b25890b19dadba0; no voters:
I20251028 14:53:19.384016 9295 leader_election.cc:290] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:19.384191 9298 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:19.384773 9298 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1b824970ab41a18b25890b19dadba0, State: Running, Role: LEADER
I20251028 14:53:19.384963 9295 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:19.385048 9298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1b824970ab41a18b25890b19dadba0" member_type: VOTER }
I20251028 14:53:19.386493 9299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e1b824970ab41a18b25890b19dadba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1b824970ab41a18b25890b19dadba0" member_type: VOTER } }
I20251028 14:53:19.386704 9299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:19.387037 9300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e1b824970ab41a18b25890b19dadba0. Latest consensus state: current_term: 1 leader_uuid: "8e1b824970ab41a18b25890b19dadba0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1b824970ab41a18b25890b19dadba0" member_type: VOTER } }
I20251028 14:53:19.387184 9300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:19.387782 9306 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:19.389313 9306 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:19.390887 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:19.392643 9306 catalog_manager.cc:1357] Generated new cluster ID: eebf6c25b49c4864be8e5e3dd1bf00a0
I20251028 14:53:19.392768 9306 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:19.407780 9306 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:19.408854 9306 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:19.423475 9306 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0: Generated new TSK 0
I20251028 14:53:19.423801 9306 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:19.434350 9250 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54616:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:19.456872 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:19.459872 9322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:19.459960 9323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:19.460486 9325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:19.460528 416 server_base.cc:1047] running on GCE node
I20251028 14:53:19.460980 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:19.461066 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:19.461105 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663199461092 us; error 0 us; skew 500 ppm
I20251028 14:53:19.461310 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:19.462723 416 webserver.cc:492] Webserver started at http://127.0.104.1:42001/ using document root <none> and password file <none>
I20251028 14:53:19.463014 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:19.463099 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:19.463239 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:19.463944 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "c67dad2c355242d3b2b8300cd9249795"
format_stamp: "Formatted at 2025-10-28 14:53:19 on dist-test-slave-0v9m"
I20251028 14:53:19.467190 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:53:19.469368 9330 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:19.469823 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251028 14:53:19.469978 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "c67dad2c355242d3b2b8300cd9249795"
format_stamp: "Formatted at 2025-10-28 14:53:19 on dist-test-slave-0v9m"
I20251028 14:53:19.470117 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:19.481372 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:19.481945 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:19.483786 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:19.483912 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:19.484035 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:19.484156 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251028 14:53:19.506669 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40983
I20251028 14:53:19.506685 9399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40983 every 8 connection(s)
I20251028 14:53:19.508571 9400 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43241
I20251028 14:53:19.508728 9400 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:19.509092 9400 heartbeater.cc:507] Master 127.0.104.62:43241 requested a full tablet report, sending...
I20251028 14:53:19.510157 9250 ts_manager.cc:194] Registered new tserver with Master: c67dad2c355242d3b2b8300cd9249795 (127.0.104.1:40983)
I20251028 14:53:19.511098 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003515838s
I20251028 14:53:19.511274 9250 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54626
I20251028 14:53:19.523370 9250 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54648:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:19.538336 9364 tablet_service.cc:1505] Processing CreateTablet for tablet 34f2513eeaad4bca87b6c046d66733bb (DEFAULT_TABLE table=txn_op_dispatcher_test [id=e80ff060bebb424eb9fe95cfc0d7b2cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:19.538341 9365 tablet_service.cc:1505] Processing CreateTablet for tablet 24f1d3e972894f3598fdf0039416e776 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=e80ff060bebb424eb9fe95cfc0d7b2cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:19.538914 9364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f2513eeaad4bca87b6c046d66733bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:19.539135 9365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f1d3e972894f3598fdf0039416e776. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:19.544270 9410 tablet_bootstrap.cc:492] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795: Bootstrap starting.
I20251028 14:53:19.545662 9410 tablet_bootstrap.cc:654] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:19.547312 9410 tablet_bootstrap.cc:492] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795: No bootstrap required, opened a new log
I20251028 14:53:19.547483 9410 ts_tablet_manager.cc:1403] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251028 14:53:19.548087 9410 raft_consensus.cc:359] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.548238 9410 raft_consensus.cc:385] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:19.548312 9410 raft_consensus.cc:740] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67dad2c355242d3b2b8300cd9249795, State: Initialized, Role: FOLLOWER
I20251028 14:53:19.548504 9410 consensus_queue.cc:260] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.548629 9410 raft_consensus.cc:399] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:19.548702 9410 raft_consensus.cc:493] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:19.548796 9410 raft_consensus.cc:3060] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:19.550696 9410 raft_consensus.cc:515] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.550937 9410 leader_election.cc:304] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67dad2c355242d3b2b8300cd9249795; no voters:
I20251028 14:53:19.551393 9410 leader_election.cc:290] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:19.551614 9412 raft_consensus.cc:2804] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:19.552101 9410 ts_tablet_manager.cc:1434] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:53:19.552416 9410 tablet_bootstrap.cc:492] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: Bootstrap starting.
I20251028 14:53:19.553879 9410 tablet_bootstrap.cc:654] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:19.552122 9400 heartbeater.cc:499] Master 127.0.104.62:43241 was elected leader, sending a full tablet report...
I20251028 14:53:19.554935 9412 raft_consensus.cc:697] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 1 LEADER]: Becoming Leader. State: Replica: c67dad2c355242d3b2b8300cd9249795, State: Running, Role: LEADER
I20251028 14:53:19.555195 9412 consensus_queue.cc:237] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.555564 9410 tablet_bootstrap.cc:492] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: No bootstrap required, opened a new log
I20251028 14:53:19.555704 9410 ts_tablet_manager.cc:1403] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251028 14:53:19.556444 9410 raft_consensus.cc:359] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.556627 9410 raft_consensus.cc:385] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:19.556705 9410 raft_consensus.cc:740] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67dad2c355242d3b2b8300cd9249795, State: Initialized, Role: FOLLOWER
I20251028 14:53:19.556900 9410 consensus_queue.cc:260] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.557022 9410 raft_consensus.cc:399] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:19.557080 9410 raft_consensus.cc:493] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:19.557153 9410 raft_consensus.cc:3060] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:19.558110 9250 catalog_manager.cc:5649] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 reported cstate change: term changed from 0 to 1, leader changed from <none> to c67dad2c355242d3b2b8300cd9249795 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c67dad2c355242d3b2b8300cd9249795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:19.559890 9410 raft_consensus.cc:515] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.560096 9410 leader_election.cc:304] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67dad2c355242d3b2b8300cd9249795; no voters:
I20251028 14:53:19.560279 9410 leader_election.cc:290] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:19.560357 9412 raft_consensus.cc:2804] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:19.560626 9412 raft_consensus.cc:697] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 1 LEADER]: Becoming Leader. State: Replica: c67dad2c355242d3b2b8300cd9249795, State: Running, Role: LEADER
I20251028 14:53:19.560740 9410 ts_tablet_manager.cc:1434] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:53:19.561000 9412 consensus_queue.cc:237] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:19.563498 9250 catalog_manager.cc:5649] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 reported cstate change: term changed from 0 to 1, leader changed from <none> to c67dad2c355242d3b2b8300cd9249795 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c67dad2c355242d3b2b8300cd9249795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:19.578655 9404 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:20.449298 9250 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54658:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:20.459197 9364 tablet_service.cc:1505] Processing CreateTablet for tablet 84323a906c874439a909b79ec09d3eec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72930fc46a08428091619177e0a94da7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:20.459717 9364 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84323a906c874439a909b79ec09d3eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:20.464792 9428 tablet_bootstrap.cc:492] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: Bootstrap starting.
I20251028 14:53:20.466573 9428 tablet_bootstrap.cc:654] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:20.468289 9428 tablet_bootstrap.cc:492] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: No bootstrap required, opened a new log
I20251028 14:53:20.468456 9428 ts_tablet_manager.cc:1403] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:20.469177 9428 raft_consensus.cc:359] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:20.469422 9428 raft_consensus.cc:385] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:20.469534 9428 raft_consensus.cc:740] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67dad2c355242d3b2b8300cd9249795, State: Initialized, Role: FOLLOWER
I20251028 14:53:20.469740 9428 consensus_queue.cc:260] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:20.469904 9428 raft_consensus.cc:399] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:20.469996 9428 raft_consensus.cc:493] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:20.470074 9428 raft_consensus.cc:3060] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:20.472043 9428 raft_consensus.cc:515] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:20.472280 9428 leader_election.cc:304] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67dad2c355242d3b2b8300cd9249795; no voters:
I20251028 14:53:20.472972 9428 leader_election.cc:290] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:20.473223 9430 raft_consensus.cc:2804] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:20.473897 9428 ts_tablet_manager.cc:1434] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20251028 14:53:20.474316 9430 raft_consensus.cc:697] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 1 LEADER]: Becoming Leader. State: Replica: c67dad2c355242d3b2b8300cd9249795, State: Running, Role: LEADER
I20251028 14:53:20.474582 9430 consensus_queue.cc:237] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } }
I20251028 14:53:20.475917 9431 tablet_replica.cc:442] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67dad2c355242d3b2b8300cd9249795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } } }
I20251028 14:53:20.476004 9432 tablet_replica.cc:442] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: TxnStatusTablet state changed. Reason: New leader c67dad2c355242d3b2b8300cd9249795. Latest consensus state: current_term: 1 leader_uuid: "c67dad2c355242d3b2b8300cd9249795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } } }
I20251028 14:53:20.476331 9431 tablet_replica.cc:445] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:20.476758 9432 tablet_replica.cc:445] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:20.477360 9434 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:20.477542 9434 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:53:20.477861 9250 catalog_manager.cc:5649] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 reported cstate change: term changed from 0 to 1, leader changed from <none> to c67dad2c355242d3b2b8300cd9249795 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c67dad2c355242d3b2b8300cd9249795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67dad2c355242d3b2b8300cd9249795" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40983 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:22.629323 9443 tablet_replica.cc:333] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: stopping tablet replica
I20251028 14:53:22.629585 9443 raft_consensus.cc:2243] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:22.629793 9443 raft_consensus.cc:2272] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:22.630534 9443 ts_tablet_manager.cc:1916] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251028 14:53:22.635592 9443 ts_tablet_manager.cc:1929] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20251028 14:53:22.635736 9443 log.cc:1199] T 24f1d3e972894f3598fdf0039416e776 P c67dad2c355242d3b2b8300cd9249795: Deleting WAL directory at /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0/minicluster-data/ts-0-root/wals/24f1d3e972894f3598fdf0039416e776
W20251028 14:53:23.652556 9446 tablet_replica.cc:1306] Illegal state: The tablet is not in a running state: STOPPED: cancelling pending write operations
W20251028 14:53:23.652707 9446 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:52928: Illegal state: The tablet is not in a running state: STOPPED
I20251028 14:53:23.653007 9446 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:52928 (ReqId={client: ec545c1bcdf14ca893a565637c822ae1, seq_no=0, attempt_no=0}) took 2015 ms. Trace:
I20251028 14:53:23.653098 9446 rpcz_store.cc:276] 1028 14:53:21.637918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:21.638023 (+ 105us) service_pool.cc:225] Handling call
1028 14:53:23.652981 (+2014958us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":314,"threads_started":1,"txn-participant-registration.queue_time_us":480}
W20251028 14:53:31.630860 9404 meta_cache.cc:1015] Timed out: LookupRpc { table: 'txn_op_dispatcher_test', partition-key: (HASH (key): 0, RANGE (key): -9223372036854775808), attempt: 1 } failed: LookupRpc timed out after deadline expired: GetTableLocations RPC to 127.0.104.62:43241 timed out after 0.001s (SENT)
W20251028 14:53:31.631044 9404 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 24f1d3e972894f3598fdf0039416e776 after 121 attempt(s): LookupRpc { table: 'txn_op_dispatcher_test', partition-key: (HASH (key): 0, RANGE (key): -9223372036854775808), attempt: 1 } failed: LookupRpc timed out after deadline expired: GetTableLocations RPC to 127.0.104.62:43241 timed out after 0.001s (SENT)
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/txn_write_ops-itest.cc:2050: Failure
Value of: op_status.ToString()
Expected: has substring "STOPPED"
Actual: "Timed out: Failed to write batch of 1 ops to tablet 24f1d3e972894f3598fdf0039416e776 after 121 attempt(s): LookupRpc { table: 'txn_op_dispatcher_test', partition-key: (HASH (key): 0, RANGE (key): -9223372036854775808), attempt: 1 } failed: LookupRpc timed out after deadline expired: GetTableLocations RPC to 127.0.104.62:43241 timed out after 0.001s (SENT)"
I20251028 14:53:31.632836 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:31.652462 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:31.653102 416 tablet_replica.cc:333] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795: stopping tablet replica
I20251028 14:53:31.653420 416 raft_consensus.cc:2243] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:31.653728 416 raft_consensus.cc:2272] T 84323a906c874439a909b79ec09d3eec P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:31.654604 416 tablet_replica.cc:333] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795: stopping tablet replica
I20251028 14:53:31.654758 416 raft_consensus.cc:2243] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:31.654934 416 raft_consensus.cc:2272] T 34f2513eeaad4bca87b6c046d66733bb P c67dad2c355242d3b2b8300cd9249795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:31.670565 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:31.676988 416 master.cc:561] Master@127.0.104.62:43241 shutting down...
I20251028 14:53:31.690116 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:31.690538 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:31.690821 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8e1b824970ab41a18b25890b19dadba0: stopping tablet replica
I20251028 14:53:31.706030 416 master.cc:583] Master@127.0.104.62:43241 shutdown complete.
I20251028 14:53:31.713979 416 test_util.cc:183] -----------------------------------------------
I20251028 14:53:31.714092 416 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1761663088357144-416-0
[ FAILED ] TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted (12386 ms)
[ RUN ] TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown
I20251028 14:53:31.716490 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:39449
I20251028 14:53:31.717104 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:31.721120 9449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:31.721125 9450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.721472 416 server_base.cc:1047] running on GCE node
W20251028 14:53:31.721120 9452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.722501 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:31.722584 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:31.722620 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663211722606 us; error 0 us; skew 500 ppm
I20251028 14:53:31.722762 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:31.724402 416 webserver.cc:492] Webserver started at http://127.0.104.62:46159/ using document root <none> and password file <none>
I20251028 14:53:31.724647 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:31.724718 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:31.724830 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:31.725543 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "fdcb8a94e8f043fc852ffd72ad4145b5"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.728641 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:53:31.730908 9457 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.731768 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251028 14:53:31.731954 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "fdcb8a94e8f043fc852ffd72ad4145b5"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.732122 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:31.749730 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:31.750356 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:31.766302 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:39449
I20251028 14:53:31.766343 9518 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:39449 every 8 connection(s)
I20251028 14:53:31.767673 9519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:31.772349 9519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5: Bootstrap starting.
I20251028 14:53:31.774212 9519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:31.776553 9519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5: No bootstrap required, opened a new log
I20251028 14:53:31.777427 9519 raft_consensus.cc:359] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" member_type: VOTER }
I20251028 14:53:31.777587 9519 raft_consensus.cc:385] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:31.777666 9519 raft_consensus.cc:740] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdcb8a94e8f043fc852ffd72ad4145b5, State: Initialized, Role: FOLLOWER
I20251028 14:53:31.777908 9519 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" member_type: VOTER }
I20251028 14:53:31.778061 9519 raft_consensus.cc:399] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:31.778137 9519 raft_consensus.cc:493] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:31.778232 9519 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:31.780305 9519 raft_consensus.cc:515] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" member_type: VOTER }
I20251028 14:53:31.780547 9519 leader_election.cc:304] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdcb8a94e8f043fc852ffd72ad4145b5; no voters:
I20251028 14:53:31.781020 9519 leader_election.cc:290] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:31.781311 9522 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:31.781749 9522 raft_consensus.cc:697] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 1 LEADER]: Becoming Leader. State: Replica: fdcb8a94e8f043fc852ffd72ad4145b5, State: Running, Role: LEADER
I20251028 14:53:31.781996 9522 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" member_type: VOTER }
I20251028 14:53:31.782090 9519 sys_catalog.cc:565] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:31.782943 9524 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdcb8a94e8f043fc852ffd72ad4145b5. Latest consensus state: current_term: 1 leader_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" member_type: VOTER } }
I20251028 14:53:31.783316 9524 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:31.782972 9523 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdcb8a94e8f043fc852ffd72ad4145b5" member_type: VOTER } }
I20251028 14:53:31.784371 9523 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:31.784415 9528 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:31.786047 9528 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:31.787751 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:31.791126 9528 catalog_manager.cc:1357] Generated new cluster ID: 128e29a09a02459c8887753ce58e64df
I20251028 14:53:31.791306 9528 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:31.806315 9528 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:31.807243 9528 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:31.815140 9528 catalog_manager.cc:6022] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5: Generated new TSK 0
I20251028 14:53:31.815445 9528 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:31.820662 9474 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58194:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:31.821199 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:31.824152 9546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:31.824146 9547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.824990 416 server_base.cc:1047] running on GCE node
W20251028 14:53:31.825922 9549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.826471 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:31.826572 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:31.826612 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663211826597 us; error 0 us; skew 500 ppm
I20251028 14:53:31.826784 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:31.828357 416 webserver.cc:492] Webserver started at http://127.0.104.1:38271/ using document root <none> and password file <none>
I20251028 14:53:31.828639 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:31.828719 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:31.828860 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:31.829635 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "7a2312f40028497c9076f2368477c902"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.833030 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:53:31.835597 9554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.836047 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:53:31.836194 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "7a2312f40028497c9076f2368477c902"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.836341 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:31.850878 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:31.851593 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:31.853504 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:31.853698 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.853853 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:31.853943 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.872852 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35473
I20251028 14:53:31.872852 9623 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35473 every 8 connection(s)
I20251028 14:53:31.875412 9624 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39449
I20251028 14:53:31.875548 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251028 14:53:31.875715 9624 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:31.876250 9624 heartbeater.cc:507] Master 127.0.104.62:39449 requested a full tablet report, sending...
I20251028 14:53:31.877750 9474 ts_manager.cc:194] Registered new tserver with Master: 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473)
W20251028 14:53:31.879227 9627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:31.879230 9626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.879942 9474 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58200
I20251028 14:53:31.883282 416 server_base.cc:1047] running on GCE node
W20251028 14:53:31.883323 9629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.883808 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:31.883888 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:31.883924 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663211883910 us; error 0 us; skew 500 ppm
I20251028 14:53:31.884063 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:31.885447 416 webserver.cc:492] Webserver started at http://127.0.104.2:43389/ using document root <none> and password file <none>
I20251028 14:53:31.885718 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:31.885803 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:31.885941 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:31.886649 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-1-root/instance:
uuid: "823b2491ec3f41fda67e4f9e478635bd"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.889714 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20251028 14:53:31.892756 9634 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.893373 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251028 14:53:31.893527 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-1-root
uuid: "823b2491ec3f41fda67e4f9e478635bd"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.893688 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:31.921868 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:31.922506 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:31.924181 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:31.924278 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.924419 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:31.924559 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.946215 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35811
I20251028 14:53:31.946218 9703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35811 every 8 connection(s)
I20251028 14:53:31.948473 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251028 14:53:31.948793 9704 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39449
I20251028 14:53:31.949007 9704 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:31.949483 9704 heartbeater.cc:507] Master 127.0.104.62:39449 requested a full tablet report, sending...
I20251028 14:53:31.950654 9474 ts_manager.cc:194] Registered new tserver with Master: 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2:35811)
W20251028 14:53:31.951573 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.951911 9474 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58220
W20251028 14:53:31.954990 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.957578 416 server_base.cc:1047] running on GCE node
W20251028 14:53:31.958757 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:31.959221 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:31.959296 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:31.959333 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663211959329 us; error 0 us; skew 500 ppm
I20251028 14:53:31.959478 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:31.960716 416 webserver.cc:492] Webserver started at http://127.0.104.3:40351/ using document root <none> and password file <none>
I20251028 14:53:31.960975 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:31.961045 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:31.961159 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:31.961825 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-2-root/instance:
uuid: "7225f0a5fe1d44ababd4060349adb2bc"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.964394 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:53:31.966156 9714 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.966835 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:53:31.967072 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-2-root
uuid: "7225f0a5fe1d44ababd4060349adb2bc"
format_stamp: "Formatted at 2025-10-28 14:53:31 on dist-test-slave-0v9m"
I20251028 14:53:31.967203 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1761663088357144-416-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:31.978736 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:31.979358 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:31.981455 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:31.981535 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:31.981609 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:31.981657 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:32.005646 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:34905
I20251028 14:53:32.005673 9783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:34905 every 8 connection(s)
I20251028 14:53:32.008049 9784 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39449
I20251028 14:53:32.008265 9784 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:32.008744 9784 heartbeater.cc:507] Master 127.0.104.62:39449 requested a full tablet report, sending...
I20251028 14:53:32.010077 9474 ts_manager.cc:194] Registered new tserver with Master: 7225f0a5fe1d44ababd4060349adb2bc (127.0.104.3:34905)
I20251028 14:53:32.010281 416 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.003670744s
I20251028 14:53:32.011016 9474 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58240
I20251028 14:53:32.022957 9474 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58258:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20251028 14:53:32.023662 9474 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_op_dispatcher_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:53:32.043005 9664 tablet_service.cc:1505] Processing CreateTablet for tablet 810494f8cbe14a2bba73e60dc165cfed (DEFAULT_TABLE table=txn_op_dispatcher_test [id=547a53d1ad54400a83d316c088864650]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:32.043292 9746 tablet_service.cc:1505] Processing CreateTablet for tablet 810494f8cbe14a2bba73e60dc165cfed (DEFAULT_TABLE table=txn_op_dispatcher_test [id=547a53d1ad54400a83d316c088864650]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:32.043398 9744 tablet_service.cc:1505] Processing CreateTablet for tablet e8039eeb80d34a8393111bff6c418ad1 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=547a53d1ad54400a83d316c088864650]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:32.043581 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810494f8cbe14a2bba73e60dc165cfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.043605 9746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810494f8cbe14a2bba73e60dc165cfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.043869 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8039eeb80d34a8393111bff6c418ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.043040 9663 tablet_service.cc:1505] Processing CreateTablet for tablet e8039eeb80d34a8393111bff6c418ad1 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=547a53d1ad54400a83d316c088864650]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:32.044312 9663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8039eeb80d34a8393111bff6c418ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.050308 9584 tablet_service.cc:1505] Processing CreateTablet for tablet 810494f8cbe14a2bba73e60dc165cfed (DEFAULT_TABLE table=txn_op_dispatcher_test [id=547a53d1ad54400a83d316c088864650]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:32.050359 9583 tablet_service.cc:1505] Processing CreateTablet for tablet e8039eeb80d34a8393111bff6c418ad1 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=547a53d1ad54400a83d316c088864650]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:32.050961 9584 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810494f8cbe14a2bba73e60dc165cfed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.051261 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8039eeb80d34a8393111bff6c418ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.070493 9799 tablet_bootstrap.cc:492] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd: Bootstrap starting.
I20251028 14:53:32.070767 9798 tablet_bootstrap.cc:492] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc: Bootstrap starting.
I20251028 14:53:32.071620 9800 tablet_bootstrap.cc:492] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902: Bootstrap starting.
I20251028 14:53:32.073041 9799 tablet_bootstrap.cc:654] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.073190 9798 tablet_bootstrap.cc:654] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.073690 9800 tablet_bootstrap.cc:654] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.075373 9798 tablet_bootstrap.cc:492] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc: No bootstrap required, opened a new log
I20251028 14:53:32.075541 9798 ts_tablet_manager.cc:1403] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251028 14:53:32.075742 9799 tablet_bootstrap.cc:492] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd: No bootstrap required, opened a new log
I20251028 14:53:32.075866 9799 ts_tablet_manager.cc:1403] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251028 14:53:32.076432 9799 raft_consensus.cc:359] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.076375 9798 raft_consensus.cc:359] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.076644 9798 raft_consensus.cc:385] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.076712 9798 raft_consensus.cc:740] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7225f0a5fe1d44ababd4060349adb2bc, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.076851 9799 raft_consensus.cc:385] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.076944 9799 raft_consensus.cc:740] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823b2491ec3f41fda67e4f9e478635bd, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.076947 9798 consensus_queue.cc:260] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.077152 9799 consensus_queue.cc:260] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.078814 9784 heartbeater.cc:499] Master 127.0.104.62:39449 was elected leader, sending a full tablet report...
I20251028 14:53:32.078891 9798 ts_tablet_manager.cc:1434] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251028 14:53:32.079321 9798 tablet_bootstrap.cc:492] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc: Bootstrap starting.
I20251028 14:53:32.081421 9798 tablet_bootstrap.cc:654] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.083139 9800 tablet_bootstrap.cc:492] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902: No bootstrap required, opened a new log
I20251028 14:53:32.083366 9800 ts_tablet_manager.cc:1403] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20251028 14:53:32.084373 9800 raft_consensus.cc:359] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.084537 9800 raft_consensus.cc:385] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.084585 9800 raft_consensus.cc:740] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2312f40028497c9076f2368477c902, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.084801 9800 consensus_queue.cc:260] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.085615 9624 heartbeater.cc:499] Master 127.0.104.62:39449 was elected leader, sending a full tablet report...
I20251028 14:53:32.085602 9800 ts_tablet_manager.cc:1434] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251028 14:53:32.086436 9800 tablet_bootstrap.cc:492] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902: Bootstrap starting.
I20251028 14:53:32.088366 9800 tablet_bootstrap.cc:654] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.090579 9704 heartbeater.cc:499] Master 127.0.104.62:39449 was elected leader, sending a full tablet report...
I20251028 14:53:32.090734 9799 ts_tablet_manager.cc:1434] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20251028 14:53:32.091403 9799 tablet_bootstrap.cc:492] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd: Bootstrap starting.
I20251028 14:53:32.093631 9799 tablet_bootstrap.cc:654] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.094789 9800 tablet_bootstrap.cc:492] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902: No bootstrap required, opened a new log
I20251028 14:53:32.095108 9800 ts_tablet_manager.cc:1403] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20251028 14:53:32.096201 9800 raft_consensus.cc:359] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.096380 9800 raft_consensus.cc:385] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.096482 9800 raft_consensus.cc:740] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2312f40028497c9076f2368477c902, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.096714 9800 consensus_queue.cc:260] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.097137 9800 ts_tablet_manager.cc:1434] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251028 14:53:32.098659 9798 tablet_bootstrap.cc:492] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc: No bootstrap required, opened a new log
I20251028 14:53:32.098882 9798 ts_tablet_manager.cc:1403] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.009s
I20251028 14:53:32.099754 9798 raft_consensus.cc:359] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.099983 9798 raft_consensus.cc:385] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.100085 9798 raft_consensus.cc:740] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7225f0a5fe1d44ababd4060349adb2bc, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.100410 9798 consensus_queue.cc:260] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.100991 9798 ts_tablet_manager.cc:1434] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251028 14:53:32.102035 9799 tablet_bootstrap.cc:492] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd: No bootstrap required, opened a new log
I20251028 14:53:32.102214 9799 ts_tablet_manager.cc:1403] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20251028 14:53:32.102888 9799 raft_consensus.cc:359] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.103065 9799 raft_consensus.cc:385] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.103148 9799 raft_consensus.cc:740] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823b2491ec3f41fda67e4f9e478635bd, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.103381 9799 consensus_queue.cc:260] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.103863 9799 ts_tablet_manager.cc:1434] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251028 14:53:32.124898 9625 tablet.cc:2378] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251028 14:53:32.221675 9804 raft_consensus.cc:493] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:53:32.221943 9804 raft_consensus.cc:515] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.223519 9804 leader_election.cc:290] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473), 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2:35811)
I20251028 14:53:32.230988 9599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8039eeb80d34a8393111bff6c418ad1" candidate_uuid: "7225f0a5fe1d44ababd4060349adb2bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2312f40028497c9076f2368477c902" is_pre_election: true
I20251028 14:53:32.230988 9679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8039eeb80d34a8393111bff6c418ad1" candidate_uuid: "7225f0a5fe1d44ababd4060349adb2bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823b2491ec3f41fda67e4f9e478635bd" is_pre_election: true
I20251028 14:53:32.231408 9599 raft_consensus.cc:2468] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7225f0a5fe1d44ababd4060349adb2bc in term 0.
I20251028 14:53:32.231405 9679 raft_consensus.cc:2468] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7225f0a5fe1d44ababd4060349adb2bc in term 0.
I20251028 14:53:32.232008 9716 leader_election.cc:304] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7225f0a5fe1d44ababd4060349adb2bc, 823b2491ec3f41fda67e4f9e478635bd; no voters:
I20251028 14:53:32.232337 9804 raft_consensus.cc:2804] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:53:32.232466 9804 raft_consensus.cc:493] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:53:32.232570 9804 raft_consensus.cc:3060] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.234894 9804 raft_consensus.cc:515] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.235564 9804 leader_election.cc:290] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 1 election: Requested vote from peers 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473), 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2:35811)
I20251028 14:53:32.236321 9679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8039eeb80d34a8393111bff6c418ad1" candidate_uuid: "7225f0a5fe1d44ababd4060349adb2bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823b2491ec3f41fda67e4f9e478635bd"
I20251028 14:53:32.236248 9599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e8039eeb80d34a8393111bff6c418ad1" candidate_uuid: "7225f0a5fe1d44ababd4060349adb2bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2312f40028497c9076f2368477c902"
I20251028 14:53:32.236507 9679 raft_consensus.cc:3060] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.236593 9599 raft_consensus.cc:3060] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.239075 9679 raft_consensus.cc:2468] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7225f0a5fe1d44ababd4060349adb2bc in term 1.
I20251028 14:53:32.239077 9599 raft_consensus.cc:2468] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7225f0a5fe1d44ababd4060349adb2bc in term 1.
I20251028 14:53:32.239612 9716 leader_election.cc:304] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7225f0a5fe1d44ababd4060349adb2bc, 823b2491ec3f41fda67e4f9e478635bd; no voters:
I20251028 14:53:32.239989 9804 raft_consensus.cc:2804] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:32.240429 9804 raft_consensus.cc:697] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 LEADER]: Becoming Leader. State: Replica: 7225f0a5fe1d44ababd4060349adb2bc, State: Running, Role: LEADER
I20251028 14:53:32.240748 9804 consensus_queue.cc:237] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.244290 9474 catalog_manager.cc:5649] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7225f0a5fe1d44ababd4060349adb2bc (127.0.104.3). New cstate: current_term: 1 leader_uuid: "7225f0a5fe1d44ababd4060349adb2bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:32.285398 9805 raft_consensus.cc:493] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:53:32.285605 9805 raft_consensus.cc:515] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.286494 9805 leader_election.cc:290] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7225f0a5fe1d44ababd4060349adb2bc (127.0.104.3:34905), 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473)
I20251028 14:53:32.294728 9599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810494f8cbe14a2bba73e60dc165cfed" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2312f40028497c9076f2368477c902" is_pre_election: true
I20251028 14:53:32.294734 9759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810494f8cbe14a2bba73e60dc165cfed" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7225f0a5fe1d44ababd4060349adb2bc" is_pre_election: true
I20251028 14:53:32.295023 9599 raft_consensus.cc:2468] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 0.
I20251028 14:53:32.295135 9759 raft_consensus.cc:2468] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 0.
I20251028 14:53:32.295503 9637 leader_election.cc:304] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2312f40028497c9076f2368477c902, 823b2491ec3f41fda67e4f9e478635bd; no voters:
I20251028 14:53:32.295825 9805 raft_consensus.cc:2804] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:53:32.295920 9805 raft_consensus.cc:493] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:53:32.296008 9805 raft_consensus.cc:3060] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.298055 9805 raft_consensus.cc:515] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.298751 9805 leader_election.cc:290] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 election: Requested vote from peers 7225f0a5fe1d44ababd4060349adb2bc (127.0.104.3:34905), 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473)
I20251028 14:53:32.299345 9759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810494f8cbe14a2bba73e60dc165cfed" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7225f0a5fe1d44ababd4060349adb2bc"
I20251028 14:53:32.299448 9599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "810494f8cbe14a2bba73e60dc165cfed" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2312f40028497c9076f2368477c902"
I20251028 14:53:32.299551 9759 raft_consensus.cc:3060] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.299631 9599 raft_consensus.cc:3060] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.301913 9759 raft_consensus.cc:2468] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 1.
I20251028 14:53:32.301915 9599 raft_consensus.cc:2468] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 1.
I20251028 14:53:32.302522 9635 leader_election.cc:304] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7225f0a5fe1d44ababd4060349adb2bc, 7a2312f40028497c9076f2368477c902, 823b2491ec3f41fda67e4f9e478635bd; no voters:
I20251028 14:53:32.302870 9805 raft_consensus.cc:2804] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:32.303350 9805 raft_consensus.cc:697] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 1 LEADER]: Becoming Leader. State: Replica: 823b2491ec3f41fda67e4f9e478635bd, State: Running, Role: LEADER
I20251028 14:53:32.303674 9805 consensus_queue.cc:237] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } }
I20251028 14:53:32.307116 9474 catalog_manager.cc:5649] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2). New cstate: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } health_report { overall_health: UNKNOWN } } }
W20251028 14:53:32.338891 9789 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:32.684072 9804 consensus_queue.cc:1048] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:53:32.689818 9804 consensus_queue.cc:1048] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:53:32.837903 9474 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58262:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:53:32.838744 9474 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251028 14:53:32.847504 9583 tablet_service.cc:1505] Processing CreateTablet for tablet c3ca1669682749afbd89ee35de5014c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2493fa1829a34d7a888451255134ed9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:32.847512 9664 tablet_service.cc:1505] Processing CreateTablet for tablet c3ca1669682749afbd89ee35de5014c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2493fa1829a34d7a888451255134ed9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:32.848078 9583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ca1669682749afbd89ee35de5014c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.848066 9664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ca1669682749afbd89ee35de5014c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.848070 9744 tablet_service.cc:1505] Processing CreateTablet for tablet c3ca1669682749afbd89ee35de5014c3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2493fa1829a34d7a888451255134ed9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:32.848538 9744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ca1669682749afbd89ee35de5014c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:32.853652 9831 tablet_bootstrap.cc:492] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: Bootstrap starting.
I20251028 14:53:32.855410 9833 tablet_bootstrap.cc:492] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: Bootstrap starting.
I20251028 14:53:32.855526 9831 tablet_bootstrap.cc:654] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.857699 9831 tablet_bootstrap.cc:492] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: No bootstrap required, opened a new log
I20251028 14:53:32.857872 9831 ts_tablet_manager.cc:1403] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:32.858348 9833 tablet_bootstrap.cc:654] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.858742 9831 raft_consensus.cc:359] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.859001 9831 raft_consensus.cc:385] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.859112 9831 raft_consensus.cc:740] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823b2491ec3f41fda67e4f9e478635bd, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.859373 9831 consensus_queue.cc:260] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.859773 9805 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:32.859951 9832 tablet_bootstrap.cc:492] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: Bootstrap starting.
I20251028 14:53:32.860008 9805 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:53:32.860574 9833 tablet_bootstrap.cc:492] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: No bootstrap required, opened a new log
I20251028 14:53:32.860805 9833 ts_tablet_manager.cc:1403] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251028 14:53:32.859875 9831 ts_tablet_manager.cc:1434] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251028 14:53:32.861932 9832 tablet_bootstrap.cc:654] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:32.862183 9833 raft_consensus.cc:359] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.862497 9833 raft_consensus.cc:385] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.862615 9833 raft_consensus.cc:740] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7225f0a5fe1d44ababd4060349adb2bc, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.862857 9833 consensus_queue.cc:260] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.863246 9804 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:32.863458 9804 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:53:32.864418 9833 ts_tablet_manager.cc:1434] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251028 14:53:32.865780 9832 tablet_bootstrap.cc:492] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: No bootstrap required, opened a new log
I20251028 14:53:32.865970 9832 ts_tablet_manager.cc:1403] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20251028 14:53:32.866760 9832 raft_consensus.cc:359] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.866935 9832 raft_consensus.cc:385] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:32.867035 9832 raft_consensus.cc:740] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a2312f40028497c9076f2368477c902, State: Initialized, Role: FOLLOWER
I20251028 14:53:32.867288 9832 consensus_queue.cc:260] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.867651 9819 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:32.867781 9832 ts_tablet_manager.cc:1434] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251028 14:53:32.867851 9819 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:53:32.910812 9805 consensus_queue.cc:1048] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:53:32.915863 9838 consensus_queue.cc:1048] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:53:32.922714 9838 raft_consensus.cc:493] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251028 14:53:32.922940 9838 raft_consensus.cc:515] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.923612 9838 leader_election.cc:290] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473), 7225f0a5fe1d44ababd4060349adb2bc (127.0.104.3:34905)
I20251028 14:53:32.924073 9599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3ca1669682749afbd89ee35de5014c3" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2312f40028497c9076f2368477c902" is_pre_election: true
I20251028 14:53:32.924101 9759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3ca1669682749afbd89ee35de5014c3" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7225f0a5fe1d44ababd4060349adb2bc" is_pre_election: true
I20251028 14:53:32.924295 9599 raft_consensus.cc:2468] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 0.
I20251028 14:53:32.924341 9759 raft_consensus.cc:2468] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 0.
I20251028 14:53:32.924657 9637 leader_election.cc:304] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a2312f40028497c9076f2368477c902, 823b2491ec3f41fda67e4f9e478635bd; no voters:
I20251028 14:53:32.924950 9838 raft_consensus.cc:2804] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251028 14:53:32.925052 9838 raft_consensus.cc:493] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251028 14:53:32.925127 9838 raft_consensus.cc:3060] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.927131 9838 raft_consensus.cc:515] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.927755 9838 leader_election.cc:290] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 election: Requested vote from peers 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473), 7225f0a5fe1d44ababd4060349adb2bc (127.0.104.3:34905)
I20251028 14:53:32.928314 9759 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3ca1669682749afbd89ee35de5014c3" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7225f0a5fe1d44ababd4060349adb2bc"
I20251028 14:53:32.928314 9599 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c3ca1669682749afbd89ee35de5014c3" candidate_uuid: "823b2491ec3f41fda67e4f9e478635bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a2312f40028497c9076f2368477c902"
I20251028 14:53:32.928560 9599 raft_consensus.cc:3060] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.928568 9759 raft_consensus.cc:3060] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:32.931581 9599 raft_consensus.cc:2468] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 1.
I20251028 14:53:32.931581 9759 raft_consensus.cc:2468] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 823b2491ec3f41fda67e4f9e478635bd in term 1.
I20251028 14:53:32.932153 9635 leader_election.cc:304] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7225f0a5fe1d44ababd4060349adb2bc, 823b2491ec3f41fda67e4f9e478635bd; no voters:
I20251028 14:53:32.932438 9838 raft_consensus.cc:2804] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:32.932622 9838 raft_consensus.cc:697] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 1 LEADER]: Becoming Leader. State: Replica: 823b2491ec3f41fda67e4f9e478635bd, State: Running, Role: LEADER
I20251028 14:53:32.932937 9838 consensus_queue.cc:237] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:32.934355 9805 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: TxnStatusTablet state changed. Reason: New leader 823b2491ec3f41fda67e4f9e478635bd. Latest consensus state: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:32.934525 9805 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:32.935001 9841 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:32.935206 9841 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:53:32.935978 9474 catalog_manager.cc:5649] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2). New cstate: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } health_report { overall_health: UNKNOWN } } }
I20251028 14:53:33.204499 9759 raft_consensus.cc:1275] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Refusing update from remote peer 823b2491ec3f41fda67e4f9e478635bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:53:33.204499 9599 raft_consensus.cc:1275] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Refusing update from remote peer 823b2491ec3f41fda67e4f9e478635bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251028 14:53:33.205518 9805 consensus_queue.cc:1048] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:53:33.205794 9838 consensus_queue.cc:1048] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251028 14:53:33.209637 9819 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: TxnStatusTablet state changed. Reason: New leader 823b2491ec3f41fda67e4f9e478635bd. Latest consensus state: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:33.209944 9819 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:53:33.211179 9842 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: TxnStatusTablet state changed. Reason: New leader 823b2491ec3f41fda67e4f9e478635bd. Latest consensus state: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:33.211437 9842 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:53:33.211365 9818 mvcc.cc:204] Tried to move back new op lower bound from 7215772521279066112 to 7215772520176164864. Current Snapshot: MvccSnapshot[applied={T|T < 7215772521279066112}]
I20251028 14:53:33.213701 9805 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:33.213969 9805 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:33.214179 9805 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:33.214365 9805 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:33.227044 9842 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:33.227437 9842 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:53:33.228875 9819 tablet_replica.cc:442] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "823b2491ec3f41fda67e4f9e478635bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } } }
I20251028 14:53:33.229113 9819 tablet_replica.cc:445] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: This TxnStatusTablet replica's current role is: FOLLOWER
I20251028 14:53:34.221873 9850 tablet_replica.cc:333] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902: stopping tablet replica
I20251028 14:53:34.221999 9852 tablet_replica.cc:333] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902: stopping tablet replica
I20251028 14:53:34.222115 9853 tablet_replica.cc:333] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd: stopping tablet replica
I20251028 14:53:34.222334 9850 raft_consensus.cc:2243] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 14:53:34.222429 9852 raft_consensus.cc:2243] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 14:53:34.222325 9854 tablet_replica.cc:333] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd: stopping tablet replica
I20251028 14:53:34.222307 9853 raft_consensus.cc:2243] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:34.222723 9854 raft_consensus.cc:2243] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 14:53:34.222817 9850 raft_consensus.cc:2272] T e8039eeb80d34a8393111bff6c418ad1 P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:34.222849 9852 raft_consensus.cc:2272] T 810494f8cbe14a2bba73e60dc165cfed P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:34.222935 9854 raft_consensus.cc:2272] T e8039eeb80d34a8393111bff6c418ad1 P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:34.223284 9853 raft_consensus.cc:2272] T 810494f8cbe14a2bba73e60dc165cfed P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:34.229667 9855 tablet_replica.cc:333] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc: stopping tablet replica
I20251028 14:53:34.229913 9855 raft_consensus.cc:2243] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 14:53:34.230113 9855 raft_consensus.cc:2272] T 810494f8cbe14a2bba73e60dc165cfed P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:34.231027 9856 tablet_replica.cc:333] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc: stopping tablet replica
I20251028 14:53:34.231261 9856 raft_consensus.cc:2243] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:34.232153 9856 raft_consensus.cc:2272] T e8039eeb80d34a8393111bff6c418ad1 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Raft consensus is shut down!
W20251028 14:53:35.260392 9859 tablet_replica.cc:1306] Illegal state: The tablet is not in a running state: SHUTDOWN: cancelling pending write operations
W20251028 14:53:35.260576 9859 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:35634: Illegal state: The tablet is not in a running state: SHUTDOWN
I20251028 14:53:35.260862 9859 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:35634 (ReqId={client: 78ff0606756144afba132c09306ef76d, seq_no=0, attempt_no=0}) took 2021 ms. Trace:
I20251028 14:53:35.260949 9859 rpcz_store.cc:276] 1028 14:53:33.239527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1028 14:53:33.239618 (+ 91us) service_pool.cc:225] Handling call
1028 14:53:35.260808 (+2021190us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":348,"threads_started":1,"txn-participant-registration.queue_time_us":490}
W20251028 14:53:43.335908 9787 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 810494f8cbe14a2bba73e60dc165cfed after 123 attempt(s): Failed to write to server: (no server available): Write(tablet: 810494f8cbe14a2bba73e60dc165cfed, num_ops: 1, num_attempts: 123) passed its deadline: Illegal state: The tablet is not in a running state: SHUTDOWN
I20251028 14:53:43.348026 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:43.362841 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:43.363372 416 tablet_replica.cc:333] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902: stopping tablet replica
I20251028 14:53:43.363714 416 raft_consensus.cc:2243] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 14:53:43.363996 416 raft_consensus.cc:2272] T c3ca1669682749afbd89ee35de5014c3 P 7a2312f40028497c9076f2368477c902 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:43.387176 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:43.391821 416 tablet_server.cc:178] TabletServer@127.0.104.2:0 shutting down...
I20251028 14:53:43.399748 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:43.400302 416 tablet_replica.cc:333] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd: stopping tablet replica
I20251028 14:53:43.400502 416 raft_consensus.cc:2243] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:43.400854 416 raft_consensus.cc:2272] T c3ca1669682749afbd89ee35de5014c3 P 823b2491ec3f41fda67e4f9e478635bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:43.451715 416 txn_status_manager.cc:765] Waiting for 1 task(s) to stop
I20251028 14:53:44.846525 9877 raft_consensus.cc:493] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 823b2491ec3f41fda67e4f9e478635bd)
I20251028 14:53:44.846742 9877 raft_consensus.cc:515] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:44.847400 9877 leader_election.cc:290] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473), 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2:35811)
W20251028 14:53:44.848038 9716 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251028 14:53:44.849313 9716 leader_election.cc:336] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2:35811): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251028 14:53:44.849476 9717 leader_election.cc:336] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473): Network error: Client connection negotiation failed: client connection to 127.0.104.1:35473: connect: Connection refused (error 111)
I20251028 14:53:44.849574 9717 leader_election.cc:304] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7225f0a5fe1d44ababd4060349adb2bc; no voters: 7a2312f40028497c9076f2368477c902, 823b2491ec3f41fda67e4f9e478635bd
I20251028 14:53:44.849822 9877 raft_consensus.cc:2749] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251028 14:53:46.751735 9880 raft_consensus.cc:493] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 823b2491ec3f41fda67e4f9e478635bd)
I20251028 14:53:46.751921 9880 raft_consensus.cc:515] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a2312f40028497c9076f2368477c902" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35473 } } peers { permanent_uuid: "823b2491ec3f41fda67e4f9e478635bd" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35811 } } peers { permanent_uuid: "7225f0a5fe1d44ababd4060349adb2bc" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34905 } }
I20251028 14:53:46.752516 9880 leader_election.cc:290] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473), 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2:35811)
W20251028 14:53:46.754305 9716 leader_election.cc:336] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 823b2491ec3f41fda67e4f9e478635bd (127.0.104.2:35811): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251028 14:53:46.754473 9717 leader_election.cc:336] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7a2312f40028497c9076f2368477c902 (127.0.104.1:35473): Network error: Client connection negotiation failed: client connection to 127.0.104.1:35473: connect: Connection refused (error 111)
I20251028 14:53:46.754583 9717 leader_election.cc:304] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7225f0a5fe1d44ababd4060349adb2bc; no voters: 7a2312f40028497c9076f2368477c902, 823b2491ec3f41fda67e4f9e478635bd
I20251028 14:53:46.754791 9880 raft_consensus.cc:2749] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251028 14:53:48.442886 416 tablet_server.cc:195] TabletServer@127.0.104.2:0 shutdown complete.
I20251028 14:53:48.448467 416 tablet_server.cc:178] TabletServer@127.0.104.3:0 shutting down...
I20251028 14:53:48.457646 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:48.457949 416 tablet_replica.cc:333] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc: stopping tablet replica
I20251028 14:53:48.458108 416 raft_consensus.cc:2243] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251028 14:53:48.458350 416 raft_consensus.cc:2272] T c3ca1669682749afbd89ee35de5014c3 P 7225f0a5fe1d44ababd4060349adb2bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:48.472937 416 tablet_server.cc:195] TabletServer@127.0.104.3:0 shutdown complete.
I20251028 14:53:48.477847 416 master.cc:561] Master@127.0.104.62:39449 shutting down...
I20251028 14:53:48.489295 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:48.489604 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:48.489750 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P fdcb8a94e8f043fc852ffd72ad4145b5: stopping tablet replica
I20251028 14:53:48.504611 416 master.cc:583] Master@127.0.104.62:39449 shutdown complete.
[ OK ] TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown (16801 ms)
[ RUN ] TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag
I20251028 14:53:48.518452 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:36551
I20251028 14:53:48.519018 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:48.521636 9883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:48.521661 9884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:48.522010 9886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:48.522488 416 server_base.cc:1047] running on GCE node
I20251028 14:53:48.522766 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:48.522835 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:48.522892 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663228522872 us; error 0 us; skew 500 ppm
I20251028 14:53:48.523065 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:48.524351 416 webserver.cc:492] Webserver started at http://127.0.104.62:45969/ using document root <none> and password file <none>
I20251028 14:53:48.524614 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:48.524698 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:48.524854 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:48.525595 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "3f6ad85041684d1b9f0d95a89d50bcf9"
format_stamp: "Formatted at 2025-10-28 14:53:48 on dist-test-slave-0v9m"
I20251028 14:53:48.529343 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251028 14:53:48.531399 9891 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:48.531822 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20251028 14:53:48.531960 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "3f6ad85041684d1b9f0d95a89d50bcf9"
format_stamp: "Formatted at 2025-10-28 14:53:48 on dist-test-slave-0v9m"
I20251028 14:53:48.532092 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:48.544646 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:48.545195 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:48.560158 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:36551
I20251028 14:53:48.560186 9952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:36551 every 8 connection(s)
I20251028 14:53:48.561646 9953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:48.566203 9953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9: Bootstrap starting.
I20251028 14:53:48.567777 9953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:48.570124 9953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9: No bootstrap required, opened a new log
I20251028 14:53:48.571034 9953 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" member_type: VOTER }
I20251028 14:53:48.571225 9953 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:48.571287 9953 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6ad85041684d1b9f0d95a89d50bcf9, State: Initialized, Role: FOLLOWER
I20251028 14:53:48.571498 9953 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" member_type: VOTER }
I20251028 14:53:48.571640 9953 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:48.571696 9953 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:48.571774 9953 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:48.573865 9953 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" member_type: VOTER }
I20251028 14:53:48.574112 9953 leader_election.cc:304] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f6ad85041684d1b9f0d95a89d50bcf9; no voters:
I20251028 14:53:48.574537 9953 leader_election.cc:290] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:48.574766 9956 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:48.575271 9956 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 1 LEADER]: Becoming Leader. State: Replica: 3f6ad85041684d1b9f0d95a89d50bcf9, State: Running, Role: LEADER
I20251028 14:53:48.575331 9953 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:48.575584 9956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" member_type: VOTER }
I20251028 14:53:48.576526 9957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" member_type: VOTER } }
I20251028 14:53:48.576747 9957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:48.576681 9958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f6ad85041684d1b9f0d95a89d50bcf9. Latest consensus state: current_term: 1 leader_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f6ad85041684d1b9f0d95a89d50bcf9" member_type: VOTER } }
I20251028 14:53:48.576843 9958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:48.577996 9962 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:48.579890 9962 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:48.580960 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:48.583602 9962 catalog_manager.cc:1357] Generated new cluster ID: 89976354946643809c333328254fbb3c
I20251028 14:53:48.583743 9962 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:48.618027 9962 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:48.618877 9962 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:48.634991 9962 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9: Generated new TSK 0
I20251028 14:53:48.635301 9962 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:48.646368 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:48.649533 9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:48.649706 9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:48.650326 9908 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48640:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:53:48.652458 9953 master.cc:464] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 1; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20251028 14:53:48.662787 9984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:48.662860 416 server_base.cc:1047] running on GCE node
I20251028 14:53:48.663224 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:48.663288 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:48.663319 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663228663315 us; error 0 us; skew 500 ppm
I20251028 14:53:48.663476 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:48.664815 416 webserver.cc:492] Webserver started at http://127.0.104.1:43927/ using document root <none> and password file <none>
I20251028 14:53:48.665050 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:48.665123 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:48.665307 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:48.665983 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "7d806228473644ab86f3117d2fef0454"
format_stamp: "Formatted at 2025-10-28 14:53:48 on dist-test-slave-0v9m"
I20251028 14:53:48.668995 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251028 14:53:48.670827 9988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:48.671241 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:53:48.671357 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "7d806228473644ab86f3117d2fef0454"
format_stamp: "Formatted at 2025-10-28 14:53:48 on dist-test-slave-0v9m"
I20251028 14:53:48.671479 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:48.686486 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:48.686993 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:48.688587 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:48.688701 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:48.688802 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:48.688849 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:48.706120 10057 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40807 every 8 connection(s)
I20251028 14:53:48.706090 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40807
I20251028 14:53:48.708222 10058 heartbeater.cc:344] Connected to a master server at 127.0.104.62:36551
I20251028 14:53:48.708583 10058 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:48.708976 10058 heartbeater.cc:507] Master 127.0.104.62:36551 requested a full tablet report, sending...
I20251028 14:53:48.709843 9908 ts_manager.cc:194] Registered new tserver with Master: 7d806228473644ab86f3117d2fef0454 (127.0.104.1:40807)
I20251028 14:53:48.710012 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002375986s
I20251028 14:53:48.710920 9908 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48656
I20251028 14:53:48.722564 9908 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48676:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20251028 14:53:48.736940 10017 tablet_service.cc:1505] Processing CreateTablet for tablet 803b69c2f2544a0685f547a46afa48bb (DEFAULT_TABLE table=txn_op_dispatcher_test [id=aa8cccfa427642318bf26e7393346cf8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:48.736940 10018 tablet_service.cc:1505] Processing CreateTablet for tablet ab6cf302f5c54c888aec75d170842b8b (DEFAULT_TABLE table=txn_op_dispatcher_test [id=aa8cccfa427642318bf26e7393346cf8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:48.737578 10017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 803b69c2f2544a0685f547a46afa48bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:48.737826 10018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab6cf302f5c54c888aec75d170842b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:48.743615 10068 tablet_bootstrap.cc:492] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454: Bootstrap starting.
I20251028 14:53:48.744926 10068 tablet_bootstrap.cc:654] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:48.746549 10068 tablet_bootstrap.cc:492] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454: No bootstrap required, opened a new log
I20251028 14:53:48.746676 10068 ts_tablet_manager.cc:1403] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251028 14:53:48.747223 10068 raft_consensus.cc:359] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.747360 10068 raft_consensus.cc:385] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:48.747393 10068 raft_consensus.cc:740] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d806228473644ab86f3117d2fef0454, State: Initialized, Role: FOLLOWER
I20251028 14:53:48.747588 10068 consensus_queue.cc:260] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.747694 10068 raft_consensus.cc:399] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:48.747730 10068 raft_consensus.cc:493] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:48.747797 10068 raft_consensus.cc:3060] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:48.749769 10068 raft_consensus.cc:515] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.749974 10068 leader_election.cc:304] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d806228473644ab86f3117d2fef0454; no voters:
I20251028 14:53:48.750584 10068 leader_election.cc:290] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:48.750737 10070 raft_consensus.cc:2804] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:48.751253 10070 raft_consensus.cc:697] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 1 LEADER]: Becoming Leader. State: Replica: 7d806228473644ab86f3117d2fef0454, State: Running, Role: LEADER
I20251028 14:53:48.751457 10058 heartbeater.cc:499] Master 127.0.104.62:36551 was elected leader, sending a full tablet report...
I20251028 14:53:48.751575 10070 consensus_queue.cc:237] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.752882 10068 ts_tablet_manager.cc:1434] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20251028 14:53:48.753161 10068 tablet_bootstrap.cc:492] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454: Bootstrap starting.
I20251028 14:53:48.753994 9908 catalog_manager.cc:5649] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d806228473644ab86f3117d2fef0454 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7d806228473644ab86f3117d2fef0454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:48.754657 10068 tablet_bootstrap.cc:654] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:48.757031 10068 tablet_bootstrap.cc:492] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454: No bootstrap required, opened a new log
I20251028 14:53:48.757318 10068 ts_tablet_manager.cc:1403] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251028 14:53:48.758383 10068 raft_consensus.cc:359] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.758562 10068 raft_consensus.cc:385] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:48.758610 10068 raft_consensus.cc:740] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d806228473644ab86f3117d2fef0454, State: Initialized, Role: FOLLOWER
I20251028 14:53:48.758808 10068 consensus_queue.cc:260] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.759037 10068 raft_consensus.cc:399] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:48.759101 10068 raft_consensus.cc:493] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:48.759207 10068 raft_consensus.cc:3060] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:48.761698 10068 raft_consensus.cc:515] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.761911 10068 leader_election.cc:304] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d806228473644ab86f3117d2fef0454; no voters:
I20251028 14:53:48.762136 10068 leader_election.cc:290] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:48.762210 10071 raft_consensus.cc:2804] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:48.762318 10071 raft_consensus.cc:697] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 1 LEADER]: Becoming Leader. State: Replica: 7d806228473644ab86f3117d2fef0454, State: Running, Role: LEADER
I20251028 14:53:48.762629 10068 ts_tablet_manager.cc:1434] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:53:48.762562 10071 consensus_queue.cc:237] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:48.765992 9907 catalog_manager.cc:5649] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d806228473644ab86f3117d2fef0454 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7d806228473644ab86f3117d2fef0454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:48.779268 10063 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:49.668300 9907 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48678:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:49.678123 10018 tablet_service.cc:1505] Processing CreateTablet for tablet acdf268e9cd54fbb8d4b6b2ff8ab859f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=533a3a05ec2f40eeb11f76b239945e80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:49.678665 10018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acdf268e9cd54fbb8d4b6b2ff8ab859f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:49.683790 10086 tablet_bootstrap.cc:492] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: Bootstrap starting.
I20251028 14:53:49.685407 10086 tablet_bootstrap.cc:654] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:49.687053 10086 tablet_bootstrap.cc:492] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: No bootstrap required, opened a new log
I20251028 14:53:49.687242 10086 ts_tablet_manager.cc:1403] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251028 14:53:49.687861 10086 raft_consensus.cc:359] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:49.688035 10086 raft_consensus.cc:385] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:49.688097 10086 raft_consensus.cc:740] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d806228473644ab86f3117d2fef0454, State: Initialized, Role: FOLLOWER
I20251028 14:53:49.688323 10086 consensus_queue.cc:260] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:49.688436 10086 raft_consensus.cc:399] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:49.688486 10086 raft_consensus.cc:493] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:49.688556 10086 raft_consensus.cc:3060] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:49.690490 10086 raft_consensus.cc:515] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:49.690696 10086 leader_election.cc:304] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d806228473644ab86f3117d2fef0454; no voters:
I20251028 14:53:49.691224 10086 leader_election.cc:290] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:49.691283 10088 raft_consensus.cc:2804] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:49.691915 10088 raft_consensus.cc:697] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 1 LEADER]: Becoming Leader. State: Replica: 7d806228473644ab86f3117d2fef0454, State: Running, Role: LEADER
I20251028 14:53:49.692245 10088 consensus_queue.cc:237] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } }
I20251028 14:53:49.692844 10086 ts_tablet_manager.cc:1434] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20251028 14:53:49.693518 10090 tablet_replica.cc:442] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: TxnStatusTablet state changed. Reason: New leader 7d806228473644ab86f3117d2fef0454. Latest consensus state: current_term: 1 leader_uuid: "7d806228473644ab86f3117d2fef0454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } } }
I20251028 14:53:49.693588 10089 tablet_replica.cc:442] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d806228473644ab86f3117d2fef0454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } } }
I20251028 14:53:49.693718 10089 tablet_replica.cc:445] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:49.693663 10090 tablet_replica.cc:445] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:49.694212 10092 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:49.694748 10092 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:53:49.694978 9907 catalog_manager.cc:5649] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d806228473644ab86f3117d2fef0454 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7d806228473644ab86f3117d2fef0454" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d806228473644ab86f3117d2fef0454" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40807 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:50.359397 10097 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:53022: Not found: txn 0 not found on tablet ab6cf302f5c54c888aec75d170842b8b
W20251028 14:53:50.360134 10064 batcher.cc:441] Not found: Failed to write batch of 1 ops to tablet ab6cf302f5c54c888aec75d170842b8b after 1 attempt(s): Failed to write to server: 7d806228473644ab86f3117d2fef0454 (127.0.104.1:40807): txn 0 not found on tablet ab6cf302f5c54c888aec75d170842b8b
I20251028 14:53:50.382929 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:50.395463 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:50.395834 416 tablet_replica.cc:333] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454: stopping tablet replica
I20251028 14:53:50.396035 416 raft_consensus.cc:2243] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:50.396231 416 raft_consensus.cc:2272] T acdf268e9cd54fbb8d4b6b2ff8ab859f P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:50.397029 416 tablet_replica.cc:333] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454: stopping tablet replica
I20251028 14:53:50.397141 416 raft_consensus.cc:2243] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:50.397352 416 raft_consensus.cc:2272] T ab6cf302f5c54c888aec75d170842b8b P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:50.397807 416 tablet_replica.cc:333] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454: stopping tablet replica
I20251028 14:53:50.397910 416 raft_consensus.cc:2243] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:50.398010 416 raft_consensus.cc:2272] T 803b69c2f2544a0685f547a46afa48bb P 7d806228473644ab86f3117d2fef0454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:50.403348 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:50.408473 416 master.cc:561] Master@127.0.104.62:36551 shutting down...
I20251028 14:53:50.419653 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:50.419934 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:50.420037 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3f6ad85041684d1b9f0d95a89d50bcf9: stopping tablet replica
I20251028 14:53:50.434283 416 master.cc:583] Master@127.0.104.62:36551 shutdown complete.
[ OK ] TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag (1929 ms)
[ DISABLED ] TxnOpDispatcherITest.DISABLED_TxnMultipleSingleRowsWithServerRestart
[ RUN ] TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload
I20251028 14:53:50.447992 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:44269
I20251028 14:53:50.448551 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:50.451350 10104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:50.451364 10103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:50.451923 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:50.452697 416 server_base.cc:1047] running on GCE node
I20251028 14:53:50.453114 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:50.453173 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:50.453209 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663230453202 us; error 0 us; skew 500 ppm
I20251028 14:53:50.453495 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:50.454977 416 webserver.cc:492] Webserver started at http://127.0.104.62:41705/ using document root <none> and password file <none>
I20251028 14:53:50.455248 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:50.455317 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:50.455451 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:50.462198 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "4b97be787b3343c2949e9b8a7c8b89a3"
format_stamp: "Formatted at 2025-10-28 14:53:50 on dist-test-slave-0v9m"
I20251028 14:53:50.465363 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:53:50.467327 10111 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:50.467886 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:53:50.468113 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "4b97be787b3343c2949e9b8a7c8b89a3"
format_stamp: "Formatted at 2025-10-28 14:53:50 on dist-test-slave-0v9m"
I20251028 14:53:50.468299 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:50.479503 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:50.480089 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:50.495280 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44269
I20251028 14:53:50.495337 10172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44269 every 8 connection(s)
I20251028 14:53:50.496670 10173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:50.501092 10173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3: Bootstrap starting.
I20251028 14:53:50.502352 10173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:50.503907 10173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3: No bootstrap required, opened a new log
I20251028 14:53:50.504513 10173 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" member_type: VOTER }
I20251028 14:53:50.504647 10173 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:50.504681 10173 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b97be787b3343c2949e9b8a7c8b89a3, State: Initialized, Role: FOLLOWER
I20251028 14:53:50.504868 10173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" member_type: VOTER }
I20251028 14:53:50.505079 10173 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:50.505136 10173 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:50.505184 10173 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:50.507100 10173 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" member_type: VOTER }
I20251028 14:53:50.507297 10173 leader_election.cc:304] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b97be787b3343c2949e9b8a7c8b89a3; no voters:
I20251028 14:53:50.507755 10173 leader_election.cc:290] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:50.508039 10176 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:50.508646 10173 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:50.508756 10176 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b97be787b3343c2949e9b8a7c8b89a3, State: Running, Role: LEADER
I20251028 14:53:50.509073 10176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" member_type: VOTER }
I20251028 14:53:50.510003 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" member_type: VOTER } }
I20251028 14:53:50.510198 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:50.510013 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b97be787b3343c2949e9b8a7c8b89a3. Latest consensus state: current_term: 1 leader_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b97be787b3343c2949e9b8a7c8b89a3" member_type: VOTER } }
I20251028 14:53:50.510358 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:50.512235 10181 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:50.514477 10181 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:50.515204 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:50.518987 10181 catalog_manager.cc:1357] Generated new cluster ID: 6c58f6e82ddf4d4aa6f53be0a824ed78
I20251028 14:53:50.519097 10181 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:50.534906 10181 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:50.535735 10181 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:50.543363 10181 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3: Generated new TSK 0
I20251028 14:53:50.543637 10181 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:50.548149 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:50.551038 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:50.551085 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:50.551692 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:50.552248 416 server_base.cc:1047] running on GCE node
I20251028 14:53:50.552700 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:50.552770 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:50.552804 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663230552797 us; error 0 us; skew 500 ppm
I20251028 14:53:50.552973 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:50.554705 416 webserver.cc:492] Webserver started at http://127.0.104.1:36417/ using document root <none> and password file <none>
I20251028 14:53:50.554996 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:50.555089 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:50.555253 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:50.556288 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "1909afef5120408984173dfc7e48b3e1"
format_stamp: "Formatted at 2025-10-28 14:53:50 on dist-test-slave-0v9m"
I20251028 14:53:50.556875 10128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45314:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:50.559374 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:53:50.561105 10208 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:50.561619 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251028 14:53:50.561748 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "1909afef5120408984173dfc7e48b3e1"
format_stamp: "Formatted at 2025-10-28 14:53:50 on dist-test-slave-0v9m"
I20251028 14:53:50.561868 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:50.571208 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:50.571738 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:50.573366 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:50.573442 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:50.573531 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:50.573578 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:50.589421 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35399
I20251028 14:53:50.589433 10277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35399 every 8 connection(s)
I20251028 14:53:50.591308 10278 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44269
I20251028 14:53:50.591449 10278 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:50.591769 10278 heartbeater.cc:507] Master 127.0.104.62:44269 requested a full tablet report, sending...
I20251028 14:53:50.592572 10128 ts_manager.cc:194] Registered new tserver with Master: 1909afef5120408984173dfc7e48b3e1 (127.0.104.1:35399)
I20251028 14:53:50.592799 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002296925s
I20251028 14:53:50.593673 10128 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45322
I20251028 14:53:50.593816 416 test_util.cc:276] Using random seed: 789760788
I20251028 14:53:50.605873 10128 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45338:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20251028 14:53:50.623764 10237 tablet_service.cc:1505] Processing CreateTablet for tablet 20093873be904a1fb3e7bc0aa90d849c (DEFAULT_TABLE table=test-workload [id=1fe49930853e415a874fff407abdc654]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:50.623764 10236 tablet_service.cc:1505] Processing CreateTablet for tablet 3df4db9567bf486c8987d7d6ecc6ff15 (DEFAULT_TABLE table=test-workload [id=1fe49930853e415a874fff407abdc654]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:50.623764 10238 tablet_service.cc:1505] Processing CreateTablet for tablet 88053ba71c484247b6352c0c7e1f4b82 (DEFAULT_TABLE table=test-workload [id=1fe49930853e415a874fff407abdc654]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:50.624354 10237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20093873be904a1fb3e7bc0aa90d849c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:50.624539 10238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88053ba71c484247b6352c0c7e1f4b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:50.624722 10236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3df4db9567bf486c8987d7d6ecc6ff15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:50.630280 10288 tablet_bootstrap.cc:492] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1: Bootstrap starting.
I20251028 14:53:50.632694 10288 tablet_bootstrap.cc:654] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:50.637393 10288 tablet_bootstrap.cc:492] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1: No bootstrap required, opened a new log
I20251028 14:53:50.637581 10288 ts_tablet_manager.cc:1403] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20251028 14:53:50.638428 10288 raft_consensus.cc:359] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.638609 10288 raft_consensus.cc:385] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:50.638687 10288 raft_consensus.cc:740] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Initialized, Role: FOLLOWER
I20251028 14:53:50.638947 10288 consensus_queue.cc:260] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.639128 10288 raft_consensus.cc:399] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:50.639219 10288 raft_consensus.cc:493] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:50.639334 10288 raft_consensus.cc:3060] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:50.641407 10288 raft_consensus.cc:515] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.641625 10288 leader_election.cc:304] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1909afef5120408984173dfc7e48b3e1; no voters:
I20251028 14:53:50.642087 10288 leader_election.cc:290] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:50.642333 10290 raft_consensus.cc:2804] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:50.642777 10288 ts_tablet_manager.cc:1434] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20251028 14:53:50.642746 10278 heartbeater.cc:499] Master 127.0.104.62:44269 was elected leader, sending a full tablet report...
I20251028 14:53:50.642761 10290 raft_consensus.cc:697] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Becoming Leader. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Running, Role: LEADER
I20251028 14:53:50.643069 10288 tablet_bootstrap.cc:492] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1: Bootstrap starting.
I20251028 14:53:50.643086 10290 consensus_queue.cc:237] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.644446 10288 tablet_bootstrap.cc:654] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:50.645735 10127 catalog_manager.cc:5649] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1909afef5120408984173dfc7e48b3e1 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1909afef5120408984173dfc7e48b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:50.650867 10288 tablet_bootstrap.cc:492] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1: No bootstrap required, opened a new log
I20251028 14:53:50.651007 10288 ts_tablet_manager.cc:1403] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20251028 14:53:50.651773 10288 raft_consensus.cc:359] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.651964 10288 raft_consensus.cc:385] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:50.652081 10288 raft_consensus.cc:740] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Initialized, Role: FOLLOWER
I20251028 14:53:50.652367 10288 consensus_queue.cc:260] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.652558 10288 raft_consensus.cc:399] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:50.652626 10288 raft_consensus.cc:493] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:50.652709 10288 raft_consensus.cc:3060] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:50.654732 10288 raft_consensus.cc:515] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.654944 10288 leader_election.cc:304] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1909afef5120408984173dfc7e48b3e1; no voters:
I20251028 14:53:50.655098 10288 leader_election.cc:290] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:50.655207 10292 raft_consensus.cc:2804] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:50.655372 10292 raft_consensus.cc:697] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Becoming Leader. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Running, Role: LEADER
I20251028 14:53:50.655514 10288 ts_tablet_manager.cc:1434] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:50.655617 10292 consensus_queue.cc:237] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.655762 10288 tablet_bootstrap.cc:492] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1: Bootstrap starting.
I20251028 14:53:50.657714 10288 tablet_bootstrap.cc:654] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:50.658018 10127 catalog_manager.cc:5649] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1909afef5120408984173dfc7e48b3e1 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1909afef5120408984173dfc7e48b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:50.659408 10288 tablet_bootstrap.cc:492] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1: No bootstrap required, opened a new log
I20251028 14:53:50.659525 10288 ts_tablet_manager.cc:1403] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:50.660094 10288 raft_consensus.cc:359] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.660287 10288 raft_consensus.cc:385] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:50.660388 10288 raft_consensus.cc:740] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Initialized, Role: FOLLOWER
I20251028 14:53:50.660634 10288 consensus_queue.cc:260] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.660781 10288 raft_consensus.cc:399] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:50.660866 10288 raft_consensus.cc:493] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:50.660959 10288 raft_consensus.cc:3060] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:50.663167 10288 raft_consensus.cc:515] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.663470 10288 leader_election.cc:304] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1909afef5120408984173dfc7e48b3e1; no voters:
I20251028 14:53:50.663633 10288 leader_election.cc:290] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:50.663699 10292 raft_consensus.cc:2804] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:50.663810 10292 raft_consensus.cc:697] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Becoming Leader. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Running, Role: LEADER
I20251028 14:53:50.664034 10288 ts_tablet_manager.cc:1434] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251028 14:53:50.664009 10292 consensus_queue.cc:237] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:50.666138 10127 catalog_manager.cc:5649] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1909afef5120408984173dfc7e48b3e1 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1909afef5120408984173dfc7e48b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:50.673184 10284 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:51.576973 10127 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45348:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:51.586731 10236 tablet_service.cc:1505] Processing CreateTablet for tablet a39d8df5cb4243c6aeaac98c3226c540 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=82bcc0fa04b642d0975427874b2e7072]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:51.587160 10236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a39d8df5cb4243c6aeaac98c3226c540. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:51.592633 10308 tablet_bootstrap.cc:492] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: Bootstrap starting.
I20251028 14:53:51.594610 10308 tablet_bootstrap.cc:654] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:51.596841 10308 tablet_bootstrap.cc:492] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: No bootstrap required, opened a new log
I20251028 14:53:51.597019 10308 ts_tablet_manager.cc:1403] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251028 14:53:51.597783 10308 raft_consensus.cc:359] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:51.597926 10308 raft_consensus.cc:385] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:51.597957 10308 raft_consensus.cc:740] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Initialized, Role: FOLLOWER
I20251028 14:53:51.598131 10308 consensus_queue.cc:260] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:51.598243 10308 raft_consensus.cc:399] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:51.598299 10308 raft_consensus.cc:493] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:51.598369 10308 raft_consensus.cc:3060] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:51.600317 10308 raft_consensus.cc:515] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:51.600515 10308 leader_election.cc:304] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1909afef5120408984173dfc7e48b3e1; no voters:
I20251028 14:53:51.601058 10308 leader_election.cc:290] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:51.601219 10310 raft_consensus.cc:2804] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:51.601990 10308 ts_tablet_manager.cc:1434] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20251028 14:53:51.602288 10310 raft_consensus.cc:697] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Becoming Leader. State: Replica: 1909afef5120408984173dfc7e48b3e1, State: Running, Role: LEADER
I20251028 14:53:51.602818 10310 consensus_queue.cc:237] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } }
I20251028 14:53:51.603718 10311 tablet_replica.cc:442] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1909afef5120408984173dfc7e48b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } } }
I20251028 14:53:51.603984 10311 tablet_replica.cc:445] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:51.603938 10312 tablet_replica.cc:442] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: TxnStatusTablet state changed. Reason: New leader 1909afef5120408984173dfc7e48b3e1. Latest consensus state: current_term: 1 leader_uuid: "1909afef5120408984173dfc7e48b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } } }
I20251028 14:53:51.604172 10312 tablet_replica.cc:445] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:51.605353 10314 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:51.605221 10127 catalog_manager.cc:5649] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1909afef5120408984173dfc7e48b3e1 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1909afef5120408984173dfc7e48b3e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1909afef5120408984173dfc7e48b3e1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35399 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:51.605505 10314 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20251028 14:53:51.834221 10332 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251028 14:53:51.852618 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:51.863030 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:51.863834 416 tablet_replica.cc:333] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1: stopping tablet replica
I20251028 14:53:51.864063 416 raft_consensus.cc:2243] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:51.864305 416 raft_consensus.cc:2272] T 3df4db9567bf486c8987d7d6ecc6ff15 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:51.865203 416 tablet_replica.cc:333] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1: stopping tablet replica
I20251028 14:53:51.865417 416 raft_consensus.cc:2243] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:51.865587 416 raft_consensus.cc:2272] T a39d8df5cb4243c6aeaac98c3226c540 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:51.866374 416 tablet_replica.cc:333] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1: stopping tablet replica
I20251028 14:53:51.866497 416 raft_consensus.cc:2243] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:51.866640 416 raft_consensus.cc:2272] T 20093873be904a1fb3e7bc0aa90d849c P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:51.867273 416 tablet_replica.cc:333] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1: stopping tablet replica
I20251028 14:53:51.867389 416 raft_consensus.cc:2243] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:51.867517 416 raft_consensus.cc:2272] T 88053ba71c484247b6352c0c7e1f4b82 P 1909afef5120408984173dfc7e48b3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:51.879523 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:51.884956 416 master.cc:561] Master@127.0.104.62:44269 shutting down...
I20251028 14:53:51.896592 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:51.896903 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:51.897019 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4b97be787b3343c2949e9b8a7c8b89a3: stopping tablet replica
I20251028 14:53:51.901535 416 master.cc:583] Master@127.0.104.62:44269 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload (1465 ms)
[ RUN ] TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload
I20251028 14:53:51.913638 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:44011
I20251028 14:53:51.914218 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:51.916857 10340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:51.916869 10341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:51.917615 10343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:51.923748 416 server_base.cc:1047] running on GCE node
I20251028 14:53:51.924063 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:51.924114 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:51.924139 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663231924134 us; error 0 us; skew 500 ppm
I20251028 14:53:51.924312 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:51.925763 416 webserver.cc:492] Webserver started at http://127.0.104.62:34933/ using document root <none> and password file <none>
I20251028 14:53:51.926012 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:51.926076 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:51.926211 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:51.926879 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "32dc733788d9435e876da9a811f0ab5d"
format_stamp: "Formatted at 2025-10-28 14:53:51 on dist-test-slave-0v9m"
I20251028 14:53:51.929562 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:53:51.931185 10348 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:51.931543 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:53:51.931648 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "32dc733788d9435e876da9a811f0ab5d"
format_stamp: "Formatted at 2025-10-28 14:53:51 on dist-test-slave-0v9m"
I20251028 14:53:51.931775 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:51.940989 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:51.941627 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:51.958110 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44011
I20251028 14:53:51.958110 10409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44011 every 8 connection(s)
I20251028 14:53:51.959794 10410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:51.964372 10410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d: Bootstrap starting.
I20251028 14:53:51.965821 10410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:51.967573 10410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d: No bootstrap required, opened a new log
I20251028 14:53:51.968226 10410 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc733788d9435e876da9a811f0ab5d" member_type: VOTER }
I20251028 14:53:51.968374 10410 raft_consensus.cc:385] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:51.968410 10410 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32dc733788d9435e876da9a811f0ab5d, State: Initialized, Role: FOLLOWER
I20251028 14:53:51.968603 10410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc733788d9435e876da9a811f0ab5d" member_type: VOTER }
I20251028 14:53:51.968708 10410 raft_consensus.cc:399] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:51.968758 10410 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:51.968827 10410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:51.971071 10410 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc733788d9435e876da9a811f0ab5d" member_type: VOTER }
I20251028 14:53:51.971292 10410 leader_election.cc:304] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32dc733788d9435e876da9a811f0ab5d; no voters:
I20251028 14:53:51.971743 10410 leader_election.cc:290] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:51.971875 10413 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:51.972254 10413 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 1 LEADER]: Becoming Leader. State: Replica: 32dc733788d9435e876da9a811f0ab5d, State: Running, Role: LEADER
I20251028 14:53:51.972633 10413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc733788d9435e876da9a811f0ab5d" member_type: VOTER }
I20251028 14:53:51.973548 10413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32dc733788d9435e876da9a811f0ab5d. Latest consensus state: current_term: 1 leader_uuid: "32dc733788d9435e876da9a811f0ab5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc733788d9435e876da9a811f0ab5d" member_type: VOTER } }
I20251028 14:53:51.973626 10415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32dc733788d9435e876da9a811f0ab5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32dc733788d9435e876da9a811f0ab5d" member_type: VOTER } }
I20251028 14:53:51.973829 10413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:51.973845 10415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:51.979075 10410 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:51.981391 10417 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:51.983538 10417 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:51.985337 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:51.988509 10417 catalog_manager.cc:1357] Generated new cluster ID: 73881d1a291343b793a58d3eca559125
I20251028 14:53:51.988615 10417 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:52.035166 10417 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:52.036020 10417 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:52.043311 10417 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d: Generated new TSK 0
I20251028 14:53:52.043584 10417 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:52.051337 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251028 14:53:52.052508 10365 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39878:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251028 14:53:52.055032 10438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:52.055303 10437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:52.056020 10440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:52.056733 416 server_base.cc:1047] running on GCE node
I20251028 14:53:52.056999 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:52.057052 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:52.057082 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663232057078 us; error 0 us; skew 500 ppm
I20251028 14:53:52.057229 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:52.058543 416 webserver.cc:492] Webserver started at http://127.0.104.1:40747/ using document root <none> and password file <none>
I20251028 14:53:52.058773 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:52.058841 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:52.058967 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:52.059631 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "f65afaf6885d4977820d898cc91588b7"
format_stamp: "Formatted at 2025-10-28 14:53:52 on dist-test-slave-0v9m"
I20251028 14:53:52.062634 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251028 14:53:52.064528 10445 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:52.064990 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251028 14:53:52.065106 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "f65afaf6885d4977820d898cc91588b7"
format_stamp: "Formatted at 2025-10-28 14:53:52 on dist-test-slave-0v9m"
I20251028 14:53:52.065243 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:52.081480 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:52.082185 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:52.084941 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:52.085086 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:52.085357 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:52.085486 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:52.119172 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:43143
I20251028 14:53:52.119239 10514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:43143 every 8 connection(s)
I20251028 14:53:52.121371 10515 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44011
I20251028 14:53:52.121536 10515 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:52.121917 10515 heartbeater.cc:507] Master 127.0.104.62:44011 requested a full tablet report, sending...
I20251028 14:53:52.122907 10365 ts_manager.cc:194] Registered new tserver with Master: f65afaf6885d4977820d898cc91588b7 (127.0.104.1:43143)
I20251028 14:53:52.123883 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.003453327s
I20251028 14:53:52.124389 10365 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39894
I20251028 14:53:52.129647 416 test_util.cc:276] Using random seed: 791296617
I20251028 14:53:52.148384 10365 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39916:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20251028 14:53:52.163277 10478 tablet_service.cc:1505] Processing CreateTablet for tablet e1bdefb5441445f193c58b7c2b9a86bf (DEFAULT_TABLE table=test-workload [id=2782b161dee34dedbc7d33d20f5448c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:52.163312 10479 tablet_service.cc:1505] Processing CreateTablet for tablet edb97b228a0e4232af58c305c947b44f (DEFAULT_TABLE table=test-workload [id=2782b161dee34dedbc7d33d20f5448c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:52.163277 10480 tablet_service.cc:1505] Processing CreateTablet for tablet a8119c7a386b4c7d9f19b90a2ee4b810 (DEFAULT_TABLE table=test-workload [id=2782b161dee34dedbc7d33d20f5448c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:52.163856 10478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1bdefb5441445f193c58b7c2b9a86bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:52.164060 10480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8119c7a386b4c7d9f19b90a2ee4b810. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:52.164220 10479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb97b228a0e4232af58c305c947b44f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:52.171051 10525 tablet_bootstrap.cc:492] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7: Bootstrap starting.
I20251028 14:53:52.172825 10525 tablet_bootstrap.cc:654] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:52.178625 10525 tablet_bootstrap.cc:492] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7: No bootstrap required, opened a new log
I20251028 14:53:52.178799 10525 ts_tablet_manager.cc:1403] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20251028 14:53:52.179467 10525 raft_consensus.cc:359] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.179620 10525 raft_consensus.cc:385] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:52.179654 10525 raft_consensus.cc:740] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Initialized, Role: FOLLOWER
I20251028 14:53:52.179875 10525 consensus_queue.cc:260] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.179997 10525 raft_consensus.cc:399] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:52.180035 10525 raft_consensus.cc:493] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:52.180126 10525 raft_consensus.cc:3060] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:52.182533 10525 raft_consensus.cc:515] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.182747 10525 leader_election.cc:304] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65afaf6885d4977820d898cc91588b7; no voters:
I20251028 14:53:52.183192 10525 leader_election.cc:290] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:52.183346 10527 raft_consensus.cc:2804] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:52.183993 10515 heartbeater.cc:499] Master 127.0.104.62:44011 was elected leader, sending a full tablet report...
I20251028 14:53:52.184041 10525 ts_tablet_manager.cc:1434] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20251028 14:53:52.184366 10525 tablet_bootstrap.cc:492] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7: Bootstrap starting.
I20251028 14:53:52.185745 10525 tablet_bootstrap.cc:654] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:52.190436 10527 raft_consensus.cc:697] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Becoming Leader. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Running, Role: LEADER
I20251028 14:53:52.190851 10527 consensus_queue.cc:237] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.194133 10365 catalog_manager.cc:5649] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65afaf6885d4977820d898cc91588b7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f65afaf6885d4977820d898cc91588b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:52.198362 10525 tablet_bootstrap.cc:492] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7: No bootstrap required, opened a new log
I20251028 14:53:52.198526 10525 ts_tablet_manager.cc:1403] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20251028 14:53:52.199342 10525 raft_consensus.cc:359] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.199525 10525 raft_consensus.cc:385] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:52.199589 10525 raft_consensus.cc:740] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Initialized, Role: FOLLOWER
I20251028 14:53:52.199745 10525 consensus_queue.cc:260] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.199848 10525 raft_consensus.cc:399] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:52.199898 10525 raft_consensus.cc:493] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:52.199975 10525 raft_consensus.cc:3060] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:52.202682 10525 raft_consensus.cc:515] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.202950 10525 leader_election.cc:304] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65afaf6885d4977820d898cc91588b7; no voters:
I20251028 14:53:52.203220 10527 raft_consensus.cc:2804] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:52.203408 10527 raft_consensus.cc:697] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Becoming Leader. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Running, Role: LEADER
I20251028 14:53:52.203649 10527 consensus_queue.cc:237] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.204532 10525 leader_election.cc:290] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:52.205121 10525 ts_tablet_manager.cc:1434] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20251028 14:53:52.205696 10525 tablet_bootstrap.cc:492] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7: Bootstrap starting.
I20251028 14:53:52.206045 10365 catalog_manager.cc:5649] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65afaf6885d4977820d898cc91588b7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f65afaf6885d4977820d898cc91588b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:52.207401 10525 tablet_bootstrap.cc:654] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:52.209216 10525 tablet_bootstrap.cc:492] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7: No bootstrap required, opened a new log
I20251028 14:53:52.209429 10525 ts_tablet_manager.cc:1403] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20251028 14:53:52.210296 10525 raft_consensus.cc:359] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.210457 10525 raft_consensus.cc:385] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:52.210542 10525 raft_consensus.cc:740] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Initialized, Role: FOLLOWER
I20251028 14:53:52.210754 10525 consensus_queue.cc:260] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.210934 10525 raft_consensus.cc:399] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:52.211001 10525 raft_consensus.cc:493] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:52.211071 10525 raft_consensus.cc:3060] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:52.214098 10525 raft_consensus.cc:515] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.214363 10525 leader_election.cc:304] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65afaf6885d4977820d898cc91588b7; no voters:
I20251028 14:53:52.214538 10525 leader_election.cc:290] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:52.214619 10527 raft_consensus.cc:2804] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:52.214804 10527 raft_consensus.cc:697] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Becoming Leader. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Running, Role: LEADER
I20251028 14:53:52.215044 10527 consensus_queue.cc:237] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:52.215194 10525 ts_tablet_manager.cc:1434] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251028 14:53:52.217089 10365 catalog_manager.cc:5649] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65afaf6885d4977820d898cc91588b7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f65afaf6885d4977820d898cc91588b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:52.223469 10521 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:53.065472 10365 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:53.073585 10480 tablet_service.cc:1505] Processing CreateTablet for tablet 499a9ba71fa1480c9b777ce81548b352 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9636cb61e6e14e58871394293f397669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:53.074029 10480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 499a9ba71fa1480c9b777ce81548b352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:53.079275 10545 tablet_bootstrap.cc:492] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: Bootstrap starting.
I20251028 14:53:53.080962 10545 tablet_bootstrap.cc:654] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:53.082597 10545 tablet_bootstrap.cc:492] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: No bootstrap required, opened a new log
I20251028 14:53:53.082739 10545 ts_tablet_manager.cc:1403] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:53.083305 10545 raft_consensus.cc:359] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:53.083441 10545 raft_consensus.cc:385] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:53.083493 10545 raft_consensus.cc:740] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Initialized, Role: FOLLOWER
I20251028 14:53:53.083675 10545 consensus_queue.cc:260] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:53.083810 10545 raft_consensus.cc:399] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:53.083858 10545 raft_consensus.cc:493] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:53.083926 10545 raft_consensus.cc:3060] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:53.085963 10545 raft_consensus.cc:515] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:53.086169 10545 leader_election.cc:304] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f65afaf6885d4977820d898cc91588b7; no voters:
I20251028 14:53:53.086614 10545 leader_election.cc:290] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:53.086752 10547 raft_consensus.cc:2804] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:53.087643 10545 ts_tablet_manager.cc:1434] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:53:53.088063 10547 raft_consensus.cc:697] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Becoming Leader. State: Replica: f65afaf6885d4977820d898cc91588b7, State: Running, Role: LEADER
I20251028 14:53:53.088449 10547 consensus_queue.cc:237] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } }
I20251028 14:53:53.089676 10548 tablet_replica.cc:442] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f65afaf6885d4977820d898cc91588b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } } }
I20251028 14:53:53.089740 10549 tablet_replica.cc:442] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: TxnStatusTablet state changed. Reason: New leader f65afaf6885d4977820d898cc91588b7. Latest consensus state: current_term: 1 leader_uuid: "f65afaf6885d4977820d898cc91588b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } } }
I20251028 14:53:53.090011 10548 tablet_replica.cc:445] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:53.090047 10549 tablet_replica.cc:445] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:53.090901 10365 catalog_manager.cc:5649] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to f65afaf6885d4977820d898cc91588b7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f65afaf6885d4977820d898cc91588b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f65afaf6885d4977820d898cc91588b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 43143 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:53.092240 10551 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:53.092383 10551 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:53:54.067982 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:54.084242 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:54.084880 416 tablet_replica.cc:333] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7: stopping tablet replica
I20251028 14:53:54.085129 416 raft_consensus.cc:2243] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:54.085386 416 raft_consensus.cc:2272] T 499a9ba71fa1480c9b777ce81548b352 P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:54.086247 416 tablet_replica.cc:333] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7: stopping tablet replica
I20251028 14:53:54.086382 416 raft_consensus.cc:2243] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:54.086516 416 raft_consensus.cc:2272] T a8119c7a386b4c7d9f19b90a2ee4b810 P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:54.087273 416 tablet_replica.cc:333] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7: stopping tablet replica
I20251028 14:53:54.087399 416 raft_consensus.cc:2243] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:54.087538 416 raft_consensus.cc:2272] T e1bdefb5441445f193c58b7c2b9a86bf P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:54.088199 416 tablet_replica.cc:333] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7: stopping tablet replica
I20251028 14:53:54.088320 416 raft_consensus.cc:2243] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:54.088443 416 raft_consensus.cc:2272] T edb97b228a0e4232af58c305c947b44f P f65afaf6885d4977820d898cc91588b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:54.094494 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:54.099640 416 master.cc:561] Master@127.0.104.62:44011 shutting down...
I20251028 14:53:54.108194 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:54.108449 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:54.108542 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P 32dc733788d9435e876da9a811f0ab5d: stopping tablet replica
I20251028 14:53:54.122383 416 master.cc:583] Master@127.0.104.62:44011 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload (2220 ms)
[ RUN ] TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads
I20251028 14:53:54.133726 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:41427
I20251028 14:53:54.134279 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:54.136726 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:54.136737 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:54.137666 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:54.137666 416 server_base.cc:1047] running on GCE node
I20251028 14:53:54.138046 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:54.138106 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:54.138129 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663234138125 us; error 0 us; skew 500 ppm
I20251028 14:53:54.138276 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:54.139457 416 webserver.cc:492] Webserver started at http://127.0.104.62:44645/ using document root <none> and password file <none>
I20251028 14:53:54.139693 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:54.139753 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:54.139890 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:54.140508 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/master-0-root/instance:
uuid: "cec9efc7214140e79421ebb42f34a3f5"
format_stamp: "Formatted at 2025-10-28 14:53:54 on dist-test-slave-0v9m"
I20251028 14:53:54.143216 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:53:54.144956 10589 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:54.145421 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:53:54.145536 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/master-0-root
uuid: "cec9efc7214140e79421ebb42f34a3f5"
format_stamp: "Formatted at 2025-10-28 14:53:54 on dist-test-slave-0v9m"
I20251028 14:53:54.145665 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:54.167196 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:54.167755 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:54.183746 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41427
I20251028 14:53:54.183763 10650 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41427 every 8 connection(s)
I20251028 14:53:54.185088 10651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:54.189579 10651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5: Bootstrap starting.
I20251028 14:53:54.190819 10651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:54.192884 10651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5: No bootstrap required, opened a new log
I20251028 14:53:54.193799 10651 raft_consensus.cc:359] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9efc7214140e79421ebb42f34a3f5" member_type: VOTER }
I20251028 14:53:54.193970 10651 raft_consensus.cc:385] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:54.194008 10651 raft_consensus.cc:740] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec9efc7214140e79421ebb42f34a3f5, State: Initialized, Role: FOLLOWER
I20251028 14:53:54.194231 10651 consensus_queue.cc:260] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9efc7214140e79421ebb42f34a3f5" member_type: VOTER }
I20251028 14:53:54.194342 10651 raft_consensus.cc:399] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:54.194406 10651 raft_consensus.cc:493] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:54.194483 10651 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:54.196875 10651 raft_consensus.cc:515] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9efc7214140e79421ebb42f34a3f5" member_type: VOTER }
I20251028 14:53:54.197149 10651 leader_election.cc:304] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cec9efc7214140e79421ebb42f34a3f5; no voters:
I20251028 14:53:54.197726 10651 leader_election.cc:290] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:54.197844 10654 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:54.198298 10654 raft_consensus.cc:697] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 1 LEADER]: Becoming Leader. State: Replica: cec9efc7214140e79421ebb42f34a3f5, State: Running, Role: LEADER
I20251028 14:53:54.198539 10654 consensus_queue.cc:237] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9efc7214140e79421ebb42f34a3f5" member_type: VOTER }
I20251028 14:53:54.198653 10651 sys_catalog.cc:565] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [sys.catalog]: configured and running, proceeding with master startup.
I20251028 14:53:54.199669 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cec9efc7214140e79421ebb42f34a3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9efc7214140e79421ebb42f34a3f5" member_type: VOTER } }
I20251028 14:53:54.199682 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cec9efc7214140e79421ebb42f34a3f5. Latest consensus state: current_term: 1 leader_uuid: "cec9efc7214140e79421ebb42f34a3f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cec9efc7214140e79421ebb42f34a3f5" member_type: VOTER } }
I20251028 14:53:54.199975 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:54.199970 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [sys.catalog]: This master's current role is: LEADER
I20251028 14:53:54.200851 10661 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251028 14:53:54.202867 10661 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251028 14:53:54.203980 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251028 14:53:54.206596 10661 catalog_manager.cc:1357] Generated new cluster ID: b666c24f7fec4e5db885d811a7e8988d
I20251028 14:53:54.206717 10661 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251028 14:53:54.241889 10661 catalog_manager.cc:1380] Generated new certificate authority record
I20251028 14:53:54.242714 10661 catalog_manager.cc:1514] Loading token signing keys...
I20251028 14:53:54.254133 10661 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5: Generated new TSK 0
I20251028 14:53:54.254398 10661 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251028 14:53:54.269009 10606 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44744:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:54.269451 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251028 14:53:54.271998 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251028 14:53:54.272595 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:54.273315 416 server_base.cc:1047] running on GCE node
W20251028 14:53:54.273537 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251028 14:53:54.273856 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251028 14:53:54.273918 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251028 14:53:54.273950 416 hybrid_clock.cc:648] HybridClock initialized: now 1761663234273946 us; error 0 us; skew 500 ppm
I20251028 14:53:54.274087 416 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251028 14:53:54.275362 416 webserver.cc:492] Webserver started at http://127.0.104.1:38837/ using document root <none> and password file <none>
I20251028 14:53:54.275589 416 fs_manager.cc:362] Metadata directory not provided
I20251028 14:53:54.275657 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251028 14:53:54.275781 416 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251028 14:53:54.276407 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/ts-0-root/instance:
uuid: "d205d106feb0429b9e9978551bbcab34"
format_stamp: "Formatted at 2025-10-28 14:53:54 on dist-test-slave-0v9m"
I20251028 14:53:54.279181 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251028 14:53:54.281513 10686 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:54.282032 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251028 14:53:54.282166 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/ts-0-root
uuid: "d205d106feb0429b9e9978551bbcab34"
format_stamp: "Formatted at 2025-10-28 14:53:54 on dist-test-slave-0v9m"
I20251028 14:53:54.282290 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskCXPJAB/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1761663088357144-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251028 14:53:54.290565 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251028 14:53:54.291204 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251028 14:53:54.293187 416 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251028 14:53:54.293354 416 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:54.293450 416 ts_tablet_manager.cc:616] Registered 0 tablets
I20251028 14:53:54.293509 416 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251028 14:53:54.312019 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35583
I20251028 14:53:54.312049 10755 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35583 every 8 connection(s)
I20251028 14:53:54.313676 10756 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41427
I20251028 14:53:54.313812 10756 heartbeater.cc:461] Registering TS with master...
I20251028 14:53:54.314160 10756 heartbeater.cc:507] Master 127.0.104.62:41427 requested a full tablet report, sending...
I20251028 14:53:54.314931 10606 ts_manager.cc:194] Registered new tserver with Master: d205d106feb0429b9e9978551bbcab34 (127.0.104.1:35583)
I20251028 14:53:54.315212 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002305519s
I20251028 14:53:54.316022 10606 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44748
I20251028 14:53:54.316303 416 test_util.cc:276] Using random seed: 793483276
I20251028 14:53:54.327837 10606 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44760:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20251028 14:53:54.341846 10715 tablet_service.cc:1505] Processing CreateTablet for tablet 2c0a9a706486478a960eb3df9ad7200c (DEFAULT_TABLE table=test-workload [id=a95b311b591c47c6a30553a27d517cd9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:54.341840 10716 tablet_service.cc:1505] Processing CreateTablet for tablet df08c9d9b1194992853566e331f8e70f (DEFAULT_TABLE table=test-workload [id=a95b311b591c47c6a30553a27d517cd9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:54.342044 10714 tablet_service.cc:1505] Processing CreateTablet for tablet bfb824f0b4cb4047960fb899085ed715 (DEFAULT_TABLE table=test-workload [id=a95b311b591c47c6a30553a27d517cd9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:54.342392 10715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0a9a706486478a960eb3df9ad7200c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:54.342554 10714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb824f0b4cb4047960fb899085ed715. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:54.342784 10716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df08c9d9b1194992853566e331f8e70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:54.348831 10766 tablet_bootstrap.cc:492] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34: Bootstrap starting.
I20251028 14:53:54.350356 10766 tablet_bootstrap.cc:654] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:54.351872 10766 tablet_bootstrap.cc:492] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34: No bootstrap required, opened a new log
I20251028 14:53:54.351987 10766 ts_tablet_manager.cc:1403] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251028 14:53:54.352507 10766 raft_consensus.cc:359] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.352656 10766 raft_consensus.cc:385] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:54.352691 10766 raft_consensus.cc:740] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Initialized, Role: FOLLOWER
I20251028 14:53:54.352869 10766 consensus_queue.cc:260] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.352967 10766 raft_consensus.cc:399] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:54.353020 10766 raft_consensus.cc:493] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:54.353086 10766 raft_consensus.cc:3060] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:54.355180 10766 raft_consensus.cc:515] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.355361 10766 leader_election.cc:304] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d205d106feb0429b9e9978551bbcab34; no voters:
I20251028 14:53:54.355820 10766 leader_election.cc:290] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:54.356017 10768 raft_consensus.cc:2804] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:54.356523 10766 ts_tablet_manager.cc:1434] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20251028 14:53:54.356549 10756 heartbeater.cc:499] Master 127.0.104.62:41427 was elected leader, sending a full tablet report...
I20251028 14:53:54.356752 10766 tablet_bootstrap.cc:492] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34: Bootstrap starting.
I20251028 14:53:54.356945 10768 raft_consensus.cc:697] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Becoming Leader. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Running, Role: LEADER
I20251028 14:53:54.357198 10768 consensus_queue.cc:237] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.358279 10766 tablet_bootstrap.cc:654] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:54.359993 10606 catalog_manager.cc:5649] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d205d106feb0429b9e9978551bbcab34 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:54.360150 10766 tablet_bootstrap.cc:492] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34: No bootstrap required, opened a new log
I20251028 14:53:54.360287 10766 ts_tablet_manager.cc:1403] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251028 14:53:54.360926 10766 raft_consensus.cc:359] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.361048 10766 raft_consensus.cc:385] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:54.361102 10766 raft_consensus.cc:740] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Initialized, Role: FOLLOWER
I20251028 14:53:54.361315 10766 consensus_queue.cc:260] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.361433 10766 raft_consensus.cc:399] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:54.361490 10766 raft_consensus.cc:493] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:54.361564 10766 raft_consensus.cc:3060] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:54.364003 10766 raft_consensus.cc:515] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.364184 10766 leader_election.cc:304] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d205d106feb0429b9e9978551bbcab34; no voters:
I20251028 14:53:54.364310 10766 leader_election.cc:290] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:54.364391 10768 raft_consensus.cc:2804] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:54.364509 10768 raft_consensus.cc:697] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Becoming Leader. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Running, Role: LEADER
I20251028 14:53:54.364729 10766 ts_tablet_manager.cc:1434] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251028 14:53:54.364714 10768 consensus_queue.cc:237] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.364996 10766 tablet_bootstrap.cc:492] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34: Bootstrap starting.
I20251028 14:53:54.366596 10766 tablet_bootstrap.cc:654] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:54.366849 10606 catalog_manager.cc:5649] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d205d106feb0429b9e9978551bbcab34 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:54.368268 10766 tablet_bootstrap.cc:492] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34: No bootstrap required, opened a new log
I20251028 14:53:54.368491 10766 ts_tablet_manager.cc:1403] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:54.369099 10766 raft_consensus.cc:359] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.369241 10766 raft_consensus.cc:385] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:54.369360 10766 raft_consensus.cc:740] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Initialized, Role: FOLLOWER
I20251028 14:53:54.369560 10766 consensus_queue.cc:260] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.369699 10766 raft_consensus.cc:399] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:54.369787 10766 raft_consensus.cc:493] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:54.369891 10766 raft_consensus.cc:3060] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:54.371759 10766 raft_consensus.cc:515] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.371968 10766 leader_election.cc:304] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d205d106feb0429b9e9978551bbcab34; no voters:
I20251028 14:53:54.372143 10766 leader_election.cc:290] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:54.372226 10768 raft_consensus.cc:2804] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:54.372400 10768 raft_consensus.cc:697] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Becoming Leader. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Running, Role: LEADER
I20251028 14:53:54.372663 10766 ts_tablet_manager.cc:1434] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251028 14:53:54.372684 10768 consensus_queue.cc:237] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:54.374855 10606 catalog_manager.cc:5649] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d205d106feb0429b9e9978551bbcab34 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } health_report { overall_health: HEALTHY } } }
W20251028 14:53:54.390256 10759 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20251028 14:53:55.282686 10606 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44768:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20251028 14:53:55.290941 10715 tablet_service.cc:1505] Processing CreateTablet for tablet 4bf7d0f227354077806d5b221d221a6e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f3db870650ac4fa49dbd93da823b2299]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251028 14:53:55.291437 10715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bf7d0f227354077806d5b221d221a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:55.296717 10786 tablet_bootstrap.cc:492] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: Bootstrap starting.
I20251028 14:53:55.298370 10786 tablet_bootstrap.cc:654] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:55.300206 10786 tablet_bootstrap.cc:492] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: No bootstrap required, opened a new log
I20251028 14:53:55.300405 10786 ts_tablet_manager.cc:1403] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:55.301102 10786 raft_consensus.cc:359] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:55.301307 10786 raft_consensus.cc:385] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:55.301388 10786 raft_consensus.cc:740] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Initialized, Role: FOLLOWER
I20251028 14:53:55.301592 10786 consensus_queue.cc:260] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:55.301704 10786 raft_consensus.cc:399] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:55.301745 10786 raft_consensus.cc:493] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:55.301796 10786 raft_consensus.cc:3060] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:55.303755 10786 raft_consensus.cc:515] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:55.303952 10786 leader_election.cc:304] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d205d106feb0429b9e9978551bbcab34; no voters:
I20251028 14:53:55.304567 10786 leader_election.cc:290] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:55.304760 10788 raft_consensus.cc:2804] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:55.305366 10786 ts_tablet_manager.cc:1434] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20251028 14:53:55.305581 10788 raft_consensus.cc:697] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Becoming Leader. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Running, Role: LEADER
I20251028 14:53:55.305866 10788 consensus_queue.cc:237] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:55.307523 10790 tablet_replica.cc:442] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: TxnStatusTablet state changed. Reason: New leader d205d106feb0429b9e9978551bbcab34. Latest consensus state: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } } }
I20251028 14:53:55.307742 10790 tablet_replica.cc:445] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:55.308544 10792 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251028 14:53:55.308686 10792 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251028 14:53:55.308758 10606 catalog_manager.cc:5649] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d205d106feb0429b9e9978551bbcab34 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:55.307559 10789 tablet_replica.cc:442] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } } }
I20251028 14:53:55.309907 10789 tablet_replica.cc:445] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: This TxnStatusTablet replica's current role is: LEADER
I20251028 14:53:56.180125 416 test_util.cc:276] Using random seed: 795347090
I20251028 14:53:56.195218 10599 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33336:
name: "default.second_table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20251028 14:53:56.205363 10714 tablet_service.cc:1505] Processing CreateTablet for tablet 611df7c3dbbf43c5af98e49c169ba554 (DEFAULT_TABLE table=default.second_table [id=68b75d568bfd42a88fa8e4a5ecf6c96f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:56.205587 10715 tablet_service.cc:1505] Processing CreateTablet for tablet 6e86032fcac3429ba1f2fb1a32ae4a3d (DEFAULT_TABLE table=default.second_table [id=68b75d568bfd42a88fa8e4a5ecf6c96f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:56.205897 10714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611df7c3dbbf43c5af98e49c169ba554. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:56.205823 10716 tablet_service.cc:1505] Processing CreateTablet for tablet b3fb662a15a2478297ff2037311c57d9 (DEFAULT_TABLE table=default.second_table [id=68b75d568bfd42a88fa8e4a5ecf6c96f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20251028 14:53:56.206120 10715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e86032fcac3429ba1f2fb1a32ae4a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:56.206327 10716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3fb662a15a2478297ff2037311c57d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251028 14:53:56.212488 10830 tablet_bootstrap.cc:492] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34: Bootstrap starting.
I20251028 14:53:56.214087 10830 tablet_bootstrap.cc:654] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:56.215876 10830 tablet_bootstrap.cc:492] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34: No bootstrap required, opened a new log
I20251028 14:53:56.216013 10830 ts_tablet_manager.cc:1403] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251028 14:53:56.216645 10830 raft_consensus.cc:359] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.216776 10830 raft_consensus.cc:385] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:56.216809 10830 raft_consensus.cc:740] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Initialized, Role: FOLLOWER
I20251028 14:53:56.216982 10830 consensus_queue.cc:260] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.217083 10830 raft_consensus.cc:399] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:56.217118 10830 raft_consensus.cc:493] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:56.217182 10830 raft_consensus.cc:3060] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:56.219235 10830 raft_consensus.cc:515] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.219480 10830 leader_election.cc:304] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d205d106feb0429b9e9978551bbcab34; no voters:
I20251028 14:53:56.219677 10830 leader_election.cc:290] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:56.219738 10820 raft_consensus.cc:2804] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:56.219915 10820 raft_consensus.cc:697] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Becoming Leader. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Running, Role: LEADER
I20251028 14:53:56.220144 10830 ts_tablet_manager.cc:1434] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251028 14:53:56.220134 10820 consensus_queue.cc:237] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.220381 10830 tablet_bootstrap.cc:492] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34: Bootstrap starting.
I20251028 14:53:56.221727 10830 tablet_bootstrap.cc:654] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:56.222524 10599 catalog_manager.cc:5649] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d205d106feb0429b9e9978551bbcab34 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:56.223526 10830 tablet_bootstrap.cc:492] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34: No bootstrap required, opened a new log
I20251028 14:53:56.223646 10830 ts_tablet_manager.cc:1403] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251028 14:53:56.224249 10830 raft_consensus.cc:359] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.224416 10830 raft_consensus.cc:385] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:56.224486 10830 raft_consensus.cc:740] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Initialized, Role: FOLLOWER
I20251028 14:53:56.224681 10830 consensus_queue.cc:260] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.224787 10830 raft_consensus.cc:399] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:56.224836 10830 raft_consensus.cc:493] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:56.224905 10830 raft_consensus.cc:3060] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:56.227293 10830 raft_consensus.cc:515] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.227540 10830 leader_election.cc:304] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d205d106feb0429b9e9978551bbcab34; no voters:
I20251028 14:53:56.227715 10830 leader_election.cc:290] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:56.227789 10820 raft_consensus.cc:2804] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:56.227983 10820 raft_consensus.cc:697] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Becoming Leader. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Running, Role: LEADER
I20251028 14:53:56.228194 10820 consensus_queue.cc:237] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.228353 10830 ts_tablet_manager.cc:1434] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251028 14:53:56.228580 10830 tablet_bootstrap.cc:492] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34: Bootstrap starting.
I20251028 14:53:56.230144 10830 tablet_bootstrap.cc:654] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34: Neither blocks nor log segments found. Creating new log.
I20251028 14:53:56.230444 10599 catalog_manager.cc:5649] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d205d106feb0429b9e9978551bbcab34 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:56.231590 10830 tablet_bootstrap.cc:492] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34: No bootstrap required, opened a new log
I20251028 14:53:56.231693 10830 ts_tablet_manager.cc:1403] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251028 14:53:56.232286 10830 raft_consensus.cc:359] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.232411 10830 raft_consensus.cc:385] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251028 14:53:56.232467 10830 raft_consensus.cc:740] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Initialized, Role: FOLLOWER
I20251028 14:53:56.232681 10830 consensus_queue.cc:260] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.232826 10830 raft_consensus.cc:399] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251028 14:53:56.232885 10830 raft_consensus.cc:493] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251028 14:53:56.232961 10830 raft_consensus.cc:3060] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 0 FOLLOWER]: Advancing to term 1
I20251028 14:53:56.235486 10830 raft_consensus.cc:515] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.235697 10830 leader_election.cc:304] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d205d106feb0429b9e9978551bbcab34; no voters:
I20251028 14:53:56.235847 10830 leader_election.cc:290] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251028 14:53:56.235980 10820 raft_consensus.cc:2804] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Leader election won for term 1
I20251028 14:53:56.236133 10820 raft_consensus.cc:697] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Becoming Leader. State: Replica: d205d106feb0429b9e9978551bbcab34, State: Running, Role: LEADER
I20251028 14:53:56.236284 10830 ts_tablet_manager.cc:1434] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251028 14:53:56.236330 10820 consensus_queue.cc:237] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } }
I20251028 14:53:56.238410 10599 catalog_manager.cc:5649] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 reported cstate change: term changed from 0 to 1, leader changed from <none> to d205d106feb0429b9e9978551bbcab34 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d205d106feb0429b9e9978551bbcab34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d205d106feb0429b9e9978551bbcab34" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35583 } health_report { overall_health: HEALTHY } } }
I20251028 14:53:56.463179 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20251028 14:53:56.471683 416 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251028 14:53:56.472303 416 tablet_replica.cc:333] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34: stopping tablet replica
I20251028 14:53:56.472575 416 raft_consensus.cc:2243] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.472776 416 raft_consensus.cc:2272] T 2c0a9a706486478a960eb3df9ad7200c P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.473778 416 tablet_replica.cc:333] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34: stopping tablet replica
I20251028 14:53:56.473918 416 raft_consensus.cc:2243] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.474062 416 raft_consensus.cc:2272] T bfb824f0b4cb4047960fb899085ed715 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.474845 416 tablet_replica.cc:333] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34: stopping tablet replica
I20251028 14:53:56.474992 416 raft_consensus.cc:2243] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.475131 416 raft_consensus.cc:2272] T 6e86032fcac3429ba1f2fb1a32ae4a3d P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.475915 416 tablet_replica.cc:333] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34: stopping tablet replica
I20251028 14:53:56.476059 416 raft_consensus.cc:2243] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.476178 416 raft_consensus.cc:2272] T df08c9d9b1194992853566e331f8e70f P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.476804 416 tablet_replica.cc:333] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34: stopping tablet replica
I20251028 14:53:56.476912 416 raft_consensus.cc:2243] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.477010 416 raft_consensus.cc:2272] T b3fb662a15a2478297ff2037311c57d9 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.477744 416 tablet_replica.cc:333] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34: stopping tablet replica
I20251028 14:53:56.477849 416 raft_consensus.cc:2243] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.477967 416 raft_consensus.cc:2272] T 611df7c3dbbf43c5af98e49c169ba554 P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.478715 416 tablet_replica.cc:333] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34: stopping tablet replica
I20251028 14:53:56.478821 416 raft_consensus.cc:2243] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.478940 416 raft_consensus.cc:2272] T 4bf7d0f227354077806d5b221d221a6e P d205d106feb0429b9e9978551bbcab34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.496172 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20251028 14:53:56.509882 416 master.cc:561] Master@127.0.104.62:41427 shutting down...
I20251028 14:53:56.519690 416 raft_consensus.cc:2243] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 1 LEADER]: Raft consensus shutting down.
I20251028 14:53:56.519966 416 raft_consensus.cc:2272] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251028 14:53:56.520116 416 tablet_replica.cc:333] T 00000000000000000000000000000000 P cec9efc7214140e79421ebb42f34a3f5: stopping tablet replica
I20251028 14:53:56.534770 416 master.cc:583] Master@127.0.104.62:41427 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads (2413 ms)
[----------] 18 tests from TxnOpDispatcherITest (83130 ms total)
[----------] Global test environment tear-down
[==========] 26 tests from 2 test suites ran. (148112 ms total)
[ PASSED ] 25 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted
1 FAILED TEST
YOU HAVE 1 DISABLED TEST
I20251028 14:53:56.548182 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 3 messages since previous log ~11 seconds ago
I20251028 14:53:56.548297 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 8 messages since previous log ~24 seconds ago
I20251028 14:53:56.548336 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/transactions/txn_status_manager.cc:765: suppressed but not reported on 99 messages since previous log ~13 seconds ago
I20251028 14:53:56.548374 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_replica.cc:1306: suppressed but not reported on 2 messages since previous log ~4 seconds ago
I20251028 14:53:56.548455 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/master.cc:464: suppressed but not reported on 3 messages since previous log ~7 seconds ago
I20251028 14:53:56.548573 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/txn_manager_proxy_rpc.cc:150: suppressed but not reported on 6 messages since previous log ~2 seconds ago