[==========] Running 26 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 8 tests from TxnWriteOpsITest
[ RUN ] TxnWriteOpsITest.CommitTimestampPropagation
2025-01-12T20:51:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:00Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250112 20:51:00.719688 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42291
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:33351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42291
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20250112 20:51:01.069674 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:01.069999 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:01.070132 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:01.077687 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:01.077807 424 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:01.077872 424 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20250112 20:51:01.078398 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:01.078480 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:01.078577 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:01.087652 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42291
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:01.089376 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:01.090986 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:01.101436 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:01.101482 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:01.101399 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:01.102200 424 server_base.cc:1034] running on GCE node
I20250112 20:51:01.102926 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:01.104334 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:01.105552 424 hybrid_clock.cc:648] HybridClock initialized: now 1736715061105469 us; error 56 us; skew 500 ppm
I20250112 20:51:01.105855 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:01.107983 424 webserver.cc:458] Webserver started at http://127.0.104.62:45313/ using document root <none> and password file <none>
I20250112 20:51:01.108729 424 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:01.108839 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:01.109126 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:01.111747 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "68c3fecf3b864502aae3aaf44897cfd8"
format_stamp: "Formatted at 2025-01-12 20:51:01 on dist-test-slave-3gql"
I20250112 20:51:01.112478 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "68c3fecf3b864502aae3aaf44897cfd8"
format_stamp: "Formatted at 2025-01-12 20:51:01 on dist-test-slave-3gql"
I20250112 20:51:01.117385 424 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20250112 20:51:01.120486 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:01.121799 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250112 20:51:01.121937 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "68c3fecf3b864502aae3aaf44897cfd8"
format_stamp: "Formatted at 2025-01-12 20:51:01 on dist-test-slave-3gql"
I20250112 20:51:01.122128 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:01.147765 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:01.148700 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:01.149032 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:01.172070 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42291
I20250112 20:51:01.172104 499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42291 every 8 connection(s)
I20250112 20:51:01.173765 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:01.176045 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 424
I20250112 20:51:01.176285 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:01.177691 500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:01.188108 500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8: Bootstrap starting.
I20250112 20:51:01.191222 500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:01.192863 500 log.cc:826] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:01.195500 500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8: No bootstrap required, opened a new log
I20250112 20:51:01.202306 500 raft_consensus.cc:357] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c3fecf3b864502aae3aaf44897cfd8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42291 } }
I20250112 20:51:01.202724 500 raft_consensus.cc:383] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:01.202841 500 raft_consensus.cc:738] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68c3fecf3b864502aae3aaf44897cfd8, State: Initialized, Role: FOLLOWER
I20250112 20:51:01.203962 500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c3fecf3b864502aae3aaf44897cfd8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42291 } }
I20250112 20:51:01.204166 500 raft_consensus.cc:397] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:01.204293 500 raft_consensus.cc:491] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:01.204440 500 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:01.206401 500 raft_consensus.cc:513] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c3fecf3b864502aae3aaf44897cfd8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42291 } }
I20250112 20:51:01.206874 500 leader_election.cc:304] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68c3fecf3b864502aae3aaf44897cfd8; no voters:
I20250112 20:51:01.207415 500 leader_election.cc:290] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:01.207553 505 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:01.208827 505 raft_consensus.cc:695] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [term 1 LEADER]: Becoming Leader. State: Replica: 68c3fecf3b864502aae3aaf44897cfd8, State: Running, Role: LEADER
I20250112 20:51:01.209312 505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c3fecf3b864502aae3aaf44897cfd8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42291 } }
I20250112 20:51:01.209597 500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:01.212016 507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68c3fecf3b864502aae3aaf44897cfd8. Latest consensus state: current_term: 1 leader_uuid: "68c3fecf3b864502aae3aaf44897cfd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c3fecf3b864502aae3aaf44897cfd8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42291 } } }
I20250112 20:51:01.212343 507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:01.213369 506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68c3fecf3b864502aae3aaf44897cfd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68c3fecf3b864502aae3aaf44897cfd8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42291 } } }
I20250112 20:51:01.213676 506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8 [sys.catalog]: This master's current role is: LEADER
W20250112 20:51:01.223160 520 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 20:51:01.223304 520 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 20:51:01.223423 514 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:01.225708 514 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:01.235191 514 catalog_manager.cc:1348] Generated new cluster ID: a401e2071093413fa3433ebb83728d7e
I20250112 20:51:01.235311 514 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:01.259083 514 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:01.261340 514 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:01.285483 514 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 68c3fecf3b864502aae3aaf44897cfd8: Generated new TSK 0
I20250112 20:51:01.286540 514 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:01.295915 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42291
--builtin_ntp_servers=127.0.104.20:33351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:01.299429 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56926:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:01.313000 500 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20250112 20:51:01.634865 529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:01.635171 529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:01.635325 529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:01.642799 529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:01.643440 529 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:01.643592 529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:01.652410 529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:42291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:01.653977 529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:01.655625 529 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:01.666867 534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:01.666894 535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:01.666894 537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:01.667585 529 server_base.cc:1034] running on GCE node
I20250112 20:51:01.668061 529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:01.668991 529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:01.670197 529 hybrid_clock.cc:648] HybridClock initialized: now 1736715061670160 us; error 30 us; skew 500 ppm
I20250112 20:51:01.670508 529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:01.672592 529 webserver.cc:458] Webserver started at http://127.0.104.1:34307/ using document root <none> and password file <none>
I20250112 20:51:01.673743 529 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:01.673846 529 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:01.674132 529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:01.676568 529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "a12a1c577eb84d038ba1a9c3896234fd"
format_stamp: "Formatted at 2025-01-12 20:51:01 on dist-test-slave-3gql"
I20250112 20:51:01.677278 529 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "a12a1c577eb84d038ba1a9c3896234fd"
format_stamp: "Formatted at 2025-01-12 20:51:01 on dist-test-slave-3gql"
I20250112 20:51:01.682101 529 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250112 20:51:01.685197 543 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:01.686475 529 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:01.686637 529 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "a12a1c577eb84d038ba1a9c3896234fd"
format_stamp: "Formatted at 2025-01-12 20:51:01 on dist-test-slave-3gql"
I20250112 20:51:01.686825 529 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:01.704403 529 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:01.705201 529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:01.705451 529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:01.708114 529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:01.708215 529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:01.708351 529 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:01.708456 529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250112 20:51:01.755493 529 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:42595
I20250112 20:51:01.755522 662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:42595 every 8 connection(s)
I20250112 20:51:01.756739 529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:01.759054 663 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42291
I20250112 20:51:01.759258 663 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:01.759951 663 heartbeater.cc:510] Master 127.0.104.62:42291 requested a full tablet report, sending...
I20250112 20:51:01.761704 454 ts_manager.cc:194] Registered new tserver with Master: a12a1c577eb84d038ba1a9c3896234fd (127.0.104.1:42595)
I20250112 20:51:01.762235 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 529
I20250112 20:51:01.762413 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:01.763199 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:60749
I20250112 20:51:01.765826 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42291
--builtin_ntp_servers=127.0.104.20:33351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:02.090636 665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:02.090950 665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:02.091092 665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:02.098601 665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:02.099277 665 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:02.099431 665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:02.108486 665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:42291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:02.110101 665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:02.111790 665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:02.123562 671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:02.123535 673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:02.123595 670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:02.124931 665 server_base.cc:1034] running on GCE node
I20250112 20:51:02.125553 665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:02.126503 665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:02.127717 665 hybrid_clock.cc:648] HybridClock initialized: now 1736715062127662 us; error 48 us; skew 500 ppm
I20250112 20:51:02.128041 665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:02.130276 665 webserver.cc:458] Webserver started at http://127.0.104.2:43281/ using document root <none> and password file <none>
I20250112 20:51:02.131016 665 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:02.131119 665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:02.131417 665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:02.134080 665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "549b35cfdd044790b52d7ee4ff9df82e"
format_stamp: "Formatted at 2025-01-12 20:51:02 on dist-test-slave-3gql"
I20250112 20:51:02.134792 665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "549b35cfdd044790b52d7ee4ff9df82e"
format_stamp: "Formatted at 2025-01-12 20:51:02 on dist-test-slave-3gql"
I20250112 20:51:02.140460 665 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20250112 20:51:02.144018 679 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:02.145412 665 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:02.145574 665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "549b35cfdd044790b52d7ee4ff9df82e"
format_stamp: "Formatted at 2025-01-12 20:51:02 on dist-test-slave-3gql"
I20250112 20:51:02.145764 665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:02.181200 665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:02.182431 665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:02.182786 665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:02.187013 665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:02.187201 665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250112 20:51:02.187363 665 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:02.187448 665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:02.241719 665 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35605
I20250112 20:51:02.241737 798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35605 every 8 connection(s)
I20250112 20:51:02.242987 665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:02.245018 799 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42291
I20250112 20:51:02.245242 799 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:02.245371 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 665
I20250112 20:51:02.245528 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:02.245975 799 heartbeater.cc:510] Master 127.0.104.62:42291 requested a full tablet report, sending...
I20250112 20:51:02.247241 454 ts_manager.cc:194] Registered new tserver with Master: 549b35cfdd044790b52d7ee4ff9df82e (127.0.104.2:35605)
I20250112 20:51:02.248212 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:35917
I20250112 20:51:02.249100 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42291
--builtin_ntp_servers=127.0.104.20:33351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:02.326913 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56930:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:02.584115 801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:02.584414 801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:02.584527 801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:02.592475 801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:02.592696 801 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:02.592851 801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:02.602397 801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:42291
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:02.604007 801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:02.605703 801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:02.617584 811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:02.617549 812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:02.617626 814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:02.618467 801 server_base.cc:1034] running on GCE node
I20250112 20:51:02.619051 801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:02.619966 801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:02.621649 801 hybrid_clock.cc:648] HybridClock initialized: now 1736715062621131 us; error 38 us; skew 500 ppm
I20250112 20:51:02.621960 801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:02.624243 801 webserver.cc:458] Webserver started at http://127.0.104.3:38127/ using document root <none> and password file <none>
I20250112 20:51:02.624969 801 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:02.625079 801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:02.625383 801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:02.627971 801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "958a17a388954af3bb97fe162942c5c0"
format_stamp: "Formatted at 2025-01-12 20:51:02 on dist-test-slave-3gql"
I20250112 20:51:02.628741 801 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "958a17a388954af3bb97fe162942c5c0"
format_stamp: "Formatted at 2025-01-12 20:51:02 on dist-test-slave-3gql"
I20250112 20:51:02.633795 801 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250112 20:51:02.636947 820 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:02.638166 801 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250112 20:51:02.638329 801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "958a17a388954af3bb97fe162942c5c0"
format_stamp: "Formatted at 2025-01-12 20:51:02 on dist-test-slave-3gql"
I20250112 20:51:02.638518 801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:02.668336 801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:02.669243 801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:02.669512 801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:02.672288 801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:02.672405 801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:02.672601 801 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:02.672698 801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:02.720443 801 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:37491
I20250112 20:51:02.720481 939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:37491 every 8 connection(s)
I20250112 20:51:02.721766 801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:02.723857 940 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42291
I20250112 20:51:02.724059 940 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:02.724861 940 heartbeater.cc:510] Master 127.0.104.62:42291 requested a full tablet report, sending...
I20250112 20:51:02.726155 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 801
I20250112 20:51:02.726193 454 ts_manager.cc:194] Registered new tserver with Master: 958a17a388954af3bb97fe162942c5c0 (127.0.104.3:37491)
I20250112 20:51:02.726318 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.CommitTimestampPropagation.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:02.727028 454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:34047
I20250112 20:51:02.730012 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:02.750082 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56946:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:02.751035 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:02.766024 663 heartbeater.cc:502] Master 127.0.104.62:42291 was elected leader, sending a full tablet report...
I20250112 20:51:02.781306 874 tablet_service.cc:1467] Processing CreateTablet for tablet 1d12731214ba4f80a81eefc6eac28d35 (DEFAULT_TABLE table=txn_write_ops_test [id=55d1afc7a77e4d998df98930db329541]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:02.782966 874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d12731214ba4f80a81eefc6eac28d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:02.783646 733 tablet_service.cc:1467] Processing CreateTablet for tablet 7d46aa224bf8480ba16962035aea2ad3 (DEFAULT_TABLE table=txn_write_ops_test [id=55d1afc7a77e4d998df98930db329541]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:02.783859 598 tablet_service.cc:1467] Processing CreateTablet for tablet 1d12731214ba4f80a81eefc6eac28d35 (DEFAULT_TABLE table=txn_write_ops_test [id=55d1afc7a77e4d998df98930db329541]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:02.784672 597 tablet_service.cc:1467] Processing CreateTablet for tablet 7d46aa224bf8480ba16962035aea2ad3 (DEFAULT_TABLE table=txn_write_ops_test [id=55d1afc7a77e4d998df98930db329541]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:02.785135 598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d12731214ba4f80a81eefc6eac28d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:02.785362 872 tablet_service.cc:1467] Processing CreateTablet for tablet 7d46aa224bf8480ba16962035aea2ad3 (DEFAULT_TABLE table=txn_write_ops_test [id=55d1afc7a77e4d998df98930db329541]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:02.785513 597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d46aa224bf8480ba16962035aea2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:02.784757 734 tablet_service.cc:1467] Processing CreateTablet for tablet 1d12731214ba4f80a81eefc6eac28d35 (DEFAULT_TABLE table=txn_write_ops_test [id=55d1afc7a77e4d998df98930db329541]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:02.785832 872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d46aa224bf8480ba16962035aea2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:02.786383 734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d12731214ba4f80a81eefc6eac28d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:02.796754 733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d46aa224bf8480ba16962035aea2ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:02.803335 958 tablet_bootstrap.cc:492] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0: Bootstrap starting.
I20250112 20:51:02.808897 957 tablet_bootstrap.cc:492] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd: Bootstrap starting.
I20250112 20:51:02.809360 958 tablet_bootstrap.cc:654] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:02.811002 958 log.cc:826] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:02.812603 959 tablet_bootstrap.cc:492] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e: Bootstrap starting.
I20250112 20:51:02.813117 957 tablet_bootstrap.cc:654] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:02.814309 957 log.cc:826] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:02.814476 958 tablet_bootstrap.cc:492] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0: No bootstrap required, opened a new log
I20250112 20:51:02.814775 958 ts_tablet_manager.cc:1397] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20250112 20:51:02.816006 959 tablet_bootstrap.cc:654] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:02.817230 957 tablet_bootstrap.cc:492] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd: No bootstrap required, opened a new log
I20250112 20:51:02.817611 957 ts_tablet_manager.cc:1397] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20250112 20:51:02.817915 959 log.cc:826] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:02.820706 959 tablet_bootstrap.cc:492] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e: No bootstrap required, opened a new log
I20250112 20:51:02.820966 959 ts_tablet_manager.cc:1397] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20250112 20:51:02.823814 958 raft_consensus.cc:357] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.824252 958 raft_consensus.cc:383] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:02.824362 958 raft_consensus.cc:738] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958a17a388954af3bb97fe162942c5c0, State: Initialized, Role: FOLLOWER
I20250112 20:51:02.824447 957 raft_consensus.cc:357] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:02.824905 957 raft_consensus.cc:383] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:02.825011 957 raft_consensus.cc:738] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12a1c577eb84d038ba1a9c3896234fd, State: Initialized, Role: FOLLOWER
I20250112 20:51:02.825038 958 consensus_queue.cc:260] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.825716 957 consensus_queue.cc:260] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:02.827360 957 ts_tablet_manager.cc:1428] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20250112 20:51:02.827798 957 tablet_bootstrap.cc:492] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd: Bootstrap starting.
I20250112 20:51:02.829761 957 tablet_bootstrap.cc:654] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:02.831306 959 raft_consensus.cc:357] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:02.831848 959 raft_consensus.cc:383] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:02.831969 959 raft_consensus.cc:738] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549b35cfdd044790b52d7ee4ff9df82e, State: Initialized, Role: FOLLOWER
I20250112 20:51:02.832708 959 consensus_queue.cc:260] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:02.833050 940 heartbeater.cc:502] Master 127.0.104.62:42291 was elected leader, sending a full tablet report...
I20250112 20:51:02.833269 958 ts_tablet_manager.cc:1428] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20250112 20:51:02.833745 958 tablet_bootstrap.cc:492] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0: Bootstrap starting.
I20250112 20:51:02.836894 958 tablet_bootstrap.cc:654] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:02.838572 957 tablet_bootstrap.cc:492] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd: No bootstrap required, opened a new log
I20250112 20:51:02.838768 957 ts_tablet_manager.cc:1397] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20250112 20:51:02.839378 957 raft_consensus.cc:357] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.839533 957 raft_consensus.cc:383] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:02.839653 957 raft_consensus.cc:738] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12a1c577eb84d038ba1a9c3896234fd, State: Initialized, Role: FOLLOWER
I20250112 20:51:02.839867 957 consensus_queue.cc:260] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.840337 957 ts_tablet_manager.cc:1428] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250112 20:51:02.841220 799 heartbeater.cc:502] Master 127.0.104.62:42291 was elected leader, sending a full tablet report...
I20250112 20:51:02.842309 959 ts_tablet_manager.cc:1428] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e: Time spent starting tablet: real 0.021s user 0.014s sys 0.006s
I20250112 20:51:02.842831 959 tablet_bootstrap.cc:492] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e: Bootstrap starting.
I20250112 20:51:02.845324 959 tablet_bootstrap.cc:654] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:02.846159 958 tablet_bootstrap.cc:492] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0: No bootstrap required, opened a new log
I20250112 20:51:02.846304 958 ts_tablet_manager.cc:1397] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20250112 20:51:02.847379 959 tablet_bootstrap.cc:492] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e: No bootstrap required, opened a new log
I20250112 20:51:02.847501 959 ts_tablet_manager.cc:1397] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:02.847472 958 raft_consensus.cc:357] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:02.847687 958 raft_consensus.cc:383] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:02.847777 958 raft_consensus.cc:738] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958a17a388954af3bb97fe162942c5c0, State: Initialized, Role: FOLLOWER
I20250112 20:51:02.848001 958 consensus_queue.cc:260] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:02.848198 959 raft_consensus.cc:357] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.848356 959 raft_consensus.cc:383] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:02.848455 959 raft_consensus.cc:738] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549b35cfdd044790b52d7ee4ff9df82e, State: Initialized, Role: FOLLOWER
I20250112 20:51:02.848780 958 ts_tablet_manager.cc:1428] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250112 20:51:02.848698 959 consensus_queue.cc:260] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.849172 959 ts_tablet_manager.cc:1428] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:02.903337 964 raft_consensus.cc:491] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:02.903566 964 raft_consensus.cc:513] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.905321 964 leader_election.cc:290] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 549b35cfdd044790b52d7ee4ff9df82e (127.0.104.2:35605), 958a17a388954af3bb97fe162942c5c0 (127.0.104.3:37491)
I20250112 20:51:02.912894 895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d46aa224bf8480ba16962035aea2ad3" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958a17a388954af3bb97fe162942c5c0" is_pre_election: true
I20250112 20:51:02.912869 754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d46aa224bf8480ba16962035aea2ad3" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549b35cfdd044790b52d7ee4ff9df82e" is_pre_election: true
I20250112 20:51:02.913467 895 raft_consensus.cc:2463] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 0.
I20250112 20:51:02.913467 754 raft_consensus.cc:2463] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 0.
I20250112 20:51:02.914276 544 leader_election.cc:304] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549b35cfdd044790b52d7ee4ff9df82e, a12a1c577eb84d038ba1a9c3896234fd; no voters:
I20250112 20:51:02.914625 964 raft_consensus.cc:2798] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:02.914767 964 raft_consensus.cc:491] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:02.914880 964 raft_consensus.cc:3054] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:02.916918 964 raft_consensus.cc:513] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.917454 964 leader_election.cc:290] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 election: Requested vote from peers 549b35cfdd044790b52d7ee4ff9df82e (127.0.104.2:35605), 958a17a388954af3bb97fe162942c5c0 (127.0.104.3:37491)
I20250112 20:51:02.917917 754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d46aa224bf8480ba16962035aea2ad3" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549b35cfdd044790b52d7ee4ff9df82e"
I20250112 20:51:02.917987 895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d46aa224bf8480ba16962035aea2ad3" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958a17a388954af3bb97fe162942c5c0"
I20250112 20:51:02.918088 754 raft_consensus.cc:3054] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:02.918208 895 raft_consensus.cc:3054] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:02.920176 754 raft_consensus.cc:2463] T 7d46aa224bf8480ba16962035aea2ad3 P 549b35cfdd044790b52d7ee4ff9df82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 1.
I20250112 20:51:02.920305 895 raft_consensus.cc:2463] T 7d46aa224bf8480ba16962035aea2ad3 P 958a17a388954af3bb97fe162942c5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 1.
I20250112 20:51:02.920615 544 leader_election.cc:304] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549b35cfdd044790b52d7ee4ff9df82e, a12a1c577eb84d038ba1a9c3896234fd; no voters:
I20250112 20:51:02.920883 964 raft_consensus.cc:2798] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:02.921293 964 raft_consensus.cc:695] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [term 1 LEADER]: Becoming Leader. State: Replica: a12a1c577eb84d038ba1a9c3896234fd, State: Running, Role: LEADER
I20250112 20:51:02.921739 964 consensus_queue.cc:237] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:02.926940 455 catalog_manager.cc:5526] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd reported cstate change: term changed from 0 to 1, leader changed from <none> to a12a1c577eb84d038ba1a9c3896234fd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:02.973557 941 tablet.cc:2367] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:02.992002 966 raft_consensus.cc:491] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:02.992267 966 raft_consensus.cc:513] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:02.994010 966 leader_election.cc:290] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 958a17a388954af3bb97fe162942c5c0 (127.0.104.3:37491), a12a1c577eb84d038ba1a9c3896234fd (127.0.104.1:42595)
W20250112 20:51:02.995043 800 tablet.cc:2367] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:03.001468 895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12731214ba4f80a81eefc6eac28d35" candidate_uuid: "549b35cfdd044790b52d7ee4ff9df82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958a17a388954af3bb97fe162942c5c0" is_pre_election: true
I20250112 20:51:03.001708 895 raft_consensus.cc:2463] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549b35cfdd044790b52d7ee4ff9df82e in term 0.
I20250112 20:51:03.001631 618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12731214ba4f80a81eefc6eac28d35" candidate_uuid: "549b35cfdd044790b52d7ee4ff9df82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a12a1c577eb84d038ba1a9c3896234fd" is_pre_election: true
I20250112 20:51:03.002094 618 raft_consensus.cc:2463] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 549b35cfdd044790b52d7ee4ff9df82e in term 0.
I20250112 20:51:03.002214 680 leader_election.cc:304] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549b35cfdd044790b52d7ee4ff9df82e, 958a17a388954af3bb97fe162942c5c0; no voters:
I20250112 20:51:03.002632 966 raft_consensus.cc:2798] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:03.002748 966 raft_consensus.cc:491] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:03.002839 966 raft_consensus.cc:3054] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:03.004874 966 raft_consensus.cc:513] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:03.005456 966 leader_election.cc:290] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [CANDIDATE]: Term 1 election: Requested vote from peers 958a17a388954af3bb97fe162942c5c0 (127.0.104.3:37491), a12a1c577eb84d038ba1a9c3896234fd (127.0.104.1:42595)
I20250112 20:51:03.005882 895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12731214ba4f80a81eefc6eac28d35" candidate_uuid: "549b35cfdd044790b52d7ee4ff9df82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958a17a388954af3bb97fe162942c5c0"
I20250112 20:51:03.005993 618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d12731214ba4f80a81eefc6eac28d35" candidate_uuid: "549b35cfdd044790b52d7ee4ff9df82e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a12a1c577eb84d038ba1a9c3896234fd"
I20250112 20:51:03.006068 895 raft_consensus.cc:3054] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:03.006178 618 raft_consensus.cc:3054] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:03.007905 895 raft_consensus.cc:2463] T 1d12731214ba4f80a81eefc6eac28d35 P 958a17a388954af3bb97fe162942c5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549b35cfdd044790b52d7ee4ff9df82e in term 1.
I20250112 20:51:03.008248 680 leader_election.cc:304] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549b35cfdd044790b52d7ee4ff9df82e, 958a17a388954af3bb97fe162942c5c0; no voters:
I20250112 20:51:03.008481 966 raft_consensus.cc:2798] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:03.008709 618 raft_consensus.cc:2463] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 549b35cfdd044790b52d7ee4ff9df82e in term 1.
I20250112 20:51:03.008916 966 raft_consensus.cc:695] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [term 1 LEADER]: Becoming Leader. State: Replica: 549b35cfdd044790b52d7ee4ff9df82e, State: Running, Role: LEADER
W20250112 20:51:03.009474 664 tablet.cc:2367] T 1d12731214ba4f80a81eefc6eac28d35 P a12a1c577eb84d038ba1a9c3896234fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:03.009424 966 consensus_queue.cc:237] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } }
I20250112 20:51:03.013646 455 catalog_manager.cc:5526] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e reported cstate change: term changed from 0 to 1, leader changed from <none> to 549b35cfdd044790b52d7ee4ff9df82e (127.0.104.2). New cstate: current_term: 1 leader_uuid: "549b35cfdd044790b52d7ee4ff9df82e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:03.094646 942 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:51:03.303370 964 consensus_queue.cc:1035] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:03.310621 981 consensus_queue.cc:1035] T 7d46aa224bf8480ba16962035aea2ad3 P a12a1c577eb84d038ba1a9c3896234fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:03.340049 454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56960:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:03.340795 454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:03.347196 734 tablet_service.cc:1467] Processing CreateTablet for tablet ecb96e3f002a446e9c19664d0347de13 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06f71e34664f4a9fa40011da032c675c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:03.347023 598 tablet_service.cc:1467] Processing CreateTablet for tablet ecb96e3f002a446e9c19664d0347de13 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06f71e34664f4a9fa40011da032c675c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:03.347201 872 tablet_service.cc:1467] Processing CreateTablet for tablet ecb96e3f002a446e9c19664d0347de13 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06f71e34664f4a9fa40011da032c675c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:03.347947 734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb96e3f002a446e9c19664d0347de13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:03.347957 872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb96e3f002a446e9c19664d0347de13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:03.347963 598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecb96e3f002a446e9c19664d0347de13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:03.354207 990 tablet_bootstrap.cc:492] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: Bootstrap starting.
I20250112 20:51:03.355154 958 tablet_bootstrap.cc:492] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: Bootstrap starting.
I20250112 20:51:03.355896 990 tablet_bootstrap.cc:654] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:03.357404 958 tablet_bootstrap.cc:654] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:03.357592 990 tablet_bootstrap.cc:492] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: No bootstrap required, opened a new log
I20250112 20:51:03.357751 990 ts_tablet_manager.cc:1397] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:03.358606 990 raft_consensus.cc:357] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.358871 990 raft_consensus.cc:383] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:03.358971 990 raft_consensus.cc:738] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 549b35cfdd044790b52d7ee4ff9df82e, State: Initialized, Role: FOLLOWER
I20250112 20:51:03.359195 990 consensus_queue.cc:260] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.359519 958 tablet_bootstrap.cc:492] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: No bootstrap required, opened a new log
I20250112 20:51:03.359575 966 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.359714 958 ts_tablet_manager.cc:1397] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250112 20:51:03.359805 966 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:03.360653 990 ts_tablet_manager.cc:1428] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:03.360625 958 raft_consensus.cc:357] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.360885 958 raft_consensus.cc:383] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:03.360962 958 raft_consensus.cc:738] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958a17a388954af3bb97fe162942c5c0, State: Initialized, Role: FOLLOWER
I20250112 20:51:03.361279 958 consensus_queue.cc:260] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.361845 958 ts_tablet_manager.cc:1428] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:03.361779 963 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.362066 963 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:03.362532 991 tablet_bootstrap.cc:492] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: Bootstrap starting.
I20250112 20:51:03.364370 991 tablet_bootstrap.cc:654] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:03.365954 991 tablet_bootstrap.cc:492] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: No bootstrap required, opened a new log
I20250112 20:51:03.366076 991 ts_tablet_manager.cc:1397] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:03.366672 991 raft_consensus.cc:357] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.366860 991 raft_consensus.cc:383] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:03.366948 991 raft_consensus.cc:738] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a12a1c577eb84d038ba1a9c3896234fd, State: Initialized, Role: FOLLOWER
I20250112 20:51:03.367170 991 consensus_queue.cc:260] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.367585 981 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.367772 981 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:03.368077 991 ts_tablet_manager.cc:1428] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:03.388087 966 consensus_queue.cc:1035] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:03.393308 975 consensus_queue.cc:1035] T 1d12731214ba4f80a81eefc6eac28d35 P 549b35cfdd044790b52d7ee4ff9df82e [LEADER]: Connected to new peer: Peer: permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:03.536099 981 raft_consensus.cc:491] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:03.536310 981 raft_consensus.cc:513] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.536967 981 leader_election.cc:290] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 549b35cfdd044790b52d7ee4ff9df82e (127.0.104.2:35605), 958a17a388954af3bb97fe162942c5c0 (127.0.104.3:37491)
I20250112 20:51:03.537966 754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb96e3f002a446e9c19664d0347de13" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549b35cfdd044790b52d7ee4ff9df82e" is_pre_election: true
I20250112 20:51:03.538235 754 raft_consensus.cc:2463] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 0.
I20250112 20:51:03.538193 895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb96e3f002a446e9c19664d0347de13" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958a17a388954af3bb97fe162942c5c0" is_pre_election: true
I20250112 20:51:03.538435 895 raft_consensus.cc:2463] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 0.
I20250112 20:51:03.538712 544 leader_election.cc:304] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549b35cfdd044790b52d7ee4ff9df82e, a12a1c577eb84d038ba1a9c3896234fd; no voters:
I20250112 20:51:03.539011 981 raft_consensus.cc:2798] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:03.539113 981 raft_consensus.cc:491] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:03.539215 981 raft_consensus.cc:3054] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:03.540992 981 raft_consensus.cc:513] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.541482 981 leader_election.cc:290] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 election: Requested vote from peers 549b35cfdd044790b52d7ee4ff9df82e (127.0.104.2:35605), 958a17a388954af3bb97fe162942c5c0 (127.0.104.3:37491)
I20250112 20:51:03.541966 754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb96e3f002a446e9c19664d0347de13" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "549b35cfdd044790b52d7ee4ff9df82e"
I20250112 20:51:03.542011 895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecb96e3f002a446e9c19664d0347de13" candidate_uuid: "a12a1c577eb84d038ba1a9c3896234fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "958a17a388954af3bb97fe162942c5c0"
I20250112 20:51:03.542219 754 raft_consensus.cc:3054] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:03.542235 895 raft_consensus.cc:3054] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:03.545056 754 raft_consensus.cc:2463] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 1.
I20250112 20:51:03.545073 895 raft_consensus.cc:2463] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a12a1c577eb84d038ba1a9c3896234fd in term 1.
I20250112 20:51:03.545504 544 leader_election.cc:304] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 549b35cfdd044790b52d7ee4ff9df82e, a12a1c577eb84d038ba1a9c3896234fd; no voters:
I20250112 20:51:03.545791 981 raft_consensus.cc:2798] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:03.545948 981 raft_consensus.cc:695] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [term 1 LEADER]: Becoming Leader. State: Replica: a12a1c577eb84d038ba1a9c3896234fd, State: Running, Role: LEADER
I20250112 20:51:03.546217 981 consensus_queue.cc:237] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } }
I20250112 20:51:03.547797 971 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: TxnStatusTablet state changed. Reason: New leader a12a1c577eb84d038ba1a9c3896234fd. Latest consensus state: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.548032 971 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:03.548637 998 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:03.548864 998 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:03.549358 455 catalog_manager.cc:5526] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd reported cstate change: term changed from 0 to 1, leader changed from <none> to a12a1c577eb84d038ba1a9c3896234fd (127.0.104.1). New cstate: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:03.977658 971 consensus_queue.cc:1035] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:03.979085 966 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: TxnStatusTablet state changed. Reason: New leader a12a1c577eb84d038ba1a9c3896234fd. Latest consensus state: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.979331 966 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:03.980952 964 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.981209 964 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:03.982012 964 consensus_queue.cc:1035] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:03.982484 966 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.982687 966 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P 549b35cfdd044790b52d7ee4ff9df82e: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:03.984766 1007 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.985064 1007 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:03.985149 1004 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: TxnStatusTablet state changed. Reason: New leader a12a1c577eb84d038ba1a9c3896234fd. Latest consensus state: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:03.985378 1004 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P 958a17a388954af3bb97fe162942c5c0: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:04.078115 964 tablet_replica.cc:440] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a12a1c577eb84d038ba1a9c3896234fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a12a1c577eb84d038ba1a9c3896234fd" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42595 } } peers { permanent_uuid: "549b35cfdd044790b52d7ee4ff9df82e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35605 } } peers { permanent_uuid: "958a17a388954af3bb97fe162942c5c0" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37491 } } }
I20250112 20:51:04.078390 964 tablet_replica.cc:443] T ecb96e3f002a446e9c19664d0347de13 P a12a1c577eb84d038ba1a9c3896234fd: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:04.782464 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 529
I20250112 20:51:04.783717 658 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:04.926529 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 529
I20250112 20:51:04.952018 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 665
I20250112 20:51:04.953137 794 generic_service.cc:196] Checking for leaks (request via RPC)
W20250112 20:51:05.081728 682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:42595: connect: Connection refused (error 111)
I20250112 20:51:05.081945 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 665
I20250112 20:51:05.110785 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 801
I20250112 20:51:05.111932 935 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:05.204898 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 801
I20250112 20:51:05.222813 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 424
I20250112 20:51:05.223846 495 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:05.333521 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 424
2025-01-12T20:51:05Z chronyd exiting
[ OK ] TxnWriteOpsITest.CommitTimestampPropagation (4698 ms)
[ RUN ] TxnWriteOpsITest.DeadlockPrevention
2025-01-12T20:51:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:05Z Disabled control of system clock
I20250112 20:51:05.401544 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36025
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:36025
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20250112 20:51:05.715457 1044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:05.715751 1044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:05.715837 1044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:05.723371 1044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:05.723479 1044 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:05.723516 1044 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20250112 20:51:05.723568 1044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:05.723616 1044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:05.723675 1044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:05.732584 1044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:36025
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:05.734053 1044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:05.735602 1044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:05.745688 1052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:05.745836 1050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:05.745728 1049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:05.746178 1044 server_base.cc:1034] running on GCE node
I20250112 20:51:05.747244 1044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:05.748689 1044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:05.749935 1044 hybrid_clock.cc:648] HybridClock initialized: now 1736715065749869 us; error 39 us; skew 500 ppm
I20250112 20:51:05.750308 1044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:05.756724 1044 webserver.cc:458] Webserver started at http://127.0.104.62:44609/ using document root <none> and password file <none>
I20250112 20:51:05.757532 1044 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:05.757678 1044 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:05.757995 1044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:05.760779 1044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "6f7978aaa4ee4b929eeb76287555818c"
format_stamp: "Formatted at 2025-01-12 20:51:05 on dist-test-slave-3gql"
I20250112 20:51:05.761606 1044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "6f7978aaa4ee4b929eeb76287555818c"
format_stamp: "Formatted at 2025-01-12 20:51:05 on dist-test-slave-3gql"
I20250112 20:51:05.766949 1044 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:05.770161 1058 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:05.771394 1044 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:05.771571 1044 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "6f7978aaa4ee4b929eeb76287555818c"
format_stamp: "Formatted at 2025-01-12 20:51:05 on dist-test-slave-3gql"
I20250112 20:51:05.771767 1044 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:05.809480 1044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:05.810380 1044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:05.810670 1044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:05.831615 1044 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:36025
I20250112 20:51:05.831652 1119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:36025 every 8 connection(s)
I20250112 20:51:05.832973 1044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:05.836704 1120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:05.839478 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 1044
I20250112 20:51:05.839692 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:05.848980 1120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c: Bootstrap starting.
I20250112 20:51:05.852787 1120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:05.853823 1120 log.cc:826] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:05.856228 1120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c: No bootstrap required, opened a new log
I20250112 20:51:05.861979 1120 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7978aaa4ee4b929eeb76287555818c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36025 } }
I20250112 20:51:05.862327 1120 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:05.862422 1120 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7978aaa4ee4b929eeb76287555818c, State: Initialized, Role: FOLLOWER
I20250112 20:51:05.862949 1120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7978aaa4ee4b929eeb76287555818c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36025 } }
I20250112 20:51:05.863142 1120 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:05.863251 1120 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:05.863399 1120 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:05.865327 1120 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7978aaa4ee4b929eeb76287555818c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36025 } }
I20250112 20:51:05.865744 1120 leader_election.cc:304] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f7978aaa4ee4b929eeb76287555818c; no voters:
I20250112 20:51:05.866333 1120 leader_election.cc:290] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:05.866461 1125 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:05.867095 1125 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [term 1 LEADER]: Becoming Leader. State: Replica: 6f7978aaa4ee4b929eeb76287555818c, State: Running, Role: LEADER
I20250112 20:51:05.867653 1125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7978aaa4ee4b929eeb76287555818c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36025 } }
I20250112 20:51:05.868163 1120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:05.870666 1127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f7978aaa4ee4b929eeb76287555818c. Latest consensus state: current_term: 1 leader_uuid: "6f7978aaa4ee4b929eeb76287555818c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7978aaa4ee4b929eeb76287555818c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36025 } } }
I20250112 20:51:05.870913 1127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:05.870751 1126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f7978aaa4ee4b929eeb76287555818c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f7978aaa4ee4b929eeb76287555818c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36025 } } }
I20250112 20:51:05.871096 1126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:05.872359 1131 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:05.878935 1131 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:05.890070 1131 catalog_manager.cc:1348] Generated new cluster ID: 138bf2a710bf4ad8bbc632c509f797f3
I20250112 20:51:05.890194 1131 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:05.903360 1131 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:05.905040 1131 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:05.926121 1131 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6f7978aaa4ee4b929eeb76287555818c: Generated new TSK 0
I20250112 20:51:05.927132 1131 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:05.933776 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36025
--builtin_ntp_servers=127.0.104.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:05.938583 1075 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52546:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:05.950184 1120 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20250112 20:51:06.266475 1149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:06.266760 1149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:06.266901 1149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:06.274792 1149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:06.274966 1149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:06.275108 1149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:06.284440 1149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:36025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:06.286190 1149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:06.287882 1149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:06.299003 1154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:06.299027 1155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:06.299069 1157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:06.299860 1149 server_base.cc:1034] running on GCE node
I20250112 20:51:06.300387 1149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:06.301239 1149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:06.302461 1149 hybrid_clock.cc:648] HybridClock initialized: now 1736715066302417 us; error 33 us; skew 500 ppm
I20250112 20:51:06.302752 1149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:06.304970 1149 webserver.cc:458] Webserver started at http://127.0.104.1:40387/ using document root <none> and password file <none>
I20250112 20:51:06.305680 1149 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:06.305763 1149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:06.306032 1149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:06.308599 1149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "a4401b340cbe43c0ab7e465e55d0ce5b"
format_stamp: "Formatted at 2025-01-12 20:51:06 on dist-test-slave-3gql"
I20250112 20:51:06.309314 1149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "a4401b340cbe43c0ab7e465e55d0ce5b"
format_stamp: "Formatted at 2025-01-12 20:51:06 on dist-test-slave-3gql"
I20250112 20:51:06.314250 1149 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20250112 20:51:06.317335 1163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:06.318535 1149 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20250112 20:51:06.318661 1149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "a4401b340cbe43c0ab7e465e55d0ce5b"
format_stamp: "Formatted at 2025-01-12 20:51:06 on dist-test-slave-3gql"
I20250112 20:51:06.318814 1149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:06.338564 1149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:06.339406 1149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:06.339672 1149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:06.342586 1149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:06.342698 1149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:06.342856 1149 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:06.342939 1149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:06.382128 1149 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46145
I20250112 20:51:06.382162 1282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46145 every 8 connection(s)
I20250112 20:51:06.383399 1149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:06.385445 1283 heartbeater.cc:346] Connected to a master server at 127.0.104.62:36025
I20250112 20:51:06.385653 1283 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:06.386376 1283 heartbeater.cc:510] Master 127.0.104.62:36025 requested a full tablet report, sending...
I20250112 20:51:06.388125 1075 ts_manager.cc:194] Registered new tserver with Master: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145)
I20250112 20:51:06.389604 1075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:57589
I20250112 20:51:06.392974 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 1149
I20250112 20:51:06.393162 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:06.395920 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36025
--builtin_ntp_servers=127.0.104.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:06.721329 1285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:06.721613 1285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:06.721769 1285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:06.729395 1285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:06.729594 1285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:06.729725 1285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:06.738672 1285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:36025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:06.740238 1285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:06.741981 1285 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:06.753134 1291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:06.753125 1293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:06.753129 1290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:06.754195 1285 server_base.cc:1034] running on GCE node
I20250112 20:51:06.754696 1285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:06.755596 1285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:06.756801 1285 hybrid_clock.cc:648] HybridClock initialized: now 1736715066756767 us; error 44 us; skew 500 ppm
I20250112 20:51:06.757104 1285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:06.759341 1285 webserver.cc:458] Webserver started at http://127.0.104.2:42201/ using document root <none> and password file <none>
I20250112 20:51:06.760037 1285 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:06.760138 1285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:06.760454 1285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:06.763146 1285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "52a72ff2aef448059e9ebac62bd4646f"
format_stamp: "Formatted at 2025-01-12 20:51:06 on dist-test-slave-3gql"
I20250112 20:51:06.763873 1285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "52a72ff2aef448059e9ebac62bd4646f"
format_stamp: "Formatted at 2025-01-12 20:51:06 on dist-test-slave-3gql"
I20250112 20:51:06.768997 1285 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250112 20:51:06.772167 1299 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:06.773561 1285 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:06.773705 1285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "52a72ff2aef448059e9ebac62bd4646f"
format_stamp: "Formatted at 2025-01-12 20:51:06 on dist-test-slave-3gql"
I20250112 20:51:06.773883 1285 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:06.790910 1285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:06.791682 1285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:06.791935 1285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:06.794970 1285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:06.795059 1285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:06.795190 1285 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:06.795253 1285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:06.836061 1285 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:34543
I20250112 20:51:06.836090 1418 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:34543 every 8 connection(s)
I20250112 20:51:06.837294 1285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:06.839288 1419 heartbeater.cc:346] Connected to a master server at 127.0.104.62:36025
I20250112 20:51:06.839489 1419 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:06.840224 1419 heartbeater.cc:510] Master 127.0.104.62:36025 requested a full tablet report, sending...
I20250112 20:51:06.841445 1075 ts_manager.cc:194] Registered new tserver with Master: 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543)
I20250112 20:51:06.842211 1075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:56617
I20250112 20:51:06.844204 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 1285
I20250112 20:51:06.844396 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:06.847484 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36025
--builtin_ntp_servers=127.0.104.20:35361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:06.963478 1075 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52554:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:07.158497 1421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:07.158761 1421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:07.158849 1421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:07.166064 1421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:07.166193 1421 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:07.166289 1421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:07.174913 1421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:36025
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:07.176362 1421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:07.177949 1421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:07.188493 1434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:07.188684 1432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:07.188658 1431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:07.189402 1421 server_base.cc:1034] running on GCE node
I20250112 20:51:07.189898 1421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:07.190676 1421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:07.191864 1421 hybrid_clock.cc:648] HybridClock initialized: now 1736715067191830 us; error 29 us; skew 500 ppm
I20250112 20:51:07.192130 1421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:07.194221 1421 webserver.cc:458] Webserver started at http://127.0.104.3:45415/ using document root <none> and password file <none>
I20250112 20:51:07.194866 1421 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:07.194950 1421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:07.195225 1421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:07.197611 1421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "24a44ead261e4e6aa90d3f612ec86a3b"
format_stamp: "Formatted at 2025-01-12 20:51:07 on dist-test-slave-3gql"
I20250112 20:51:07.198334 1421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "24a44ead261e4e6aa90d3f612ec86a3b"
format_stamp: "Formatted at 2025-01-12 20:51:07 on dist-test-slave-3gql"
I20250112 20:51:07.203182 1421 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250112 20:51:07.206434 1440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:07.207726 1421 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:07.207871 1421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "24a44ead261e4e6aa90d3f612ec86a3b"
format_stamp: "Formatted at 2025-01-12 20:51:07 on dist-test-slave-3gql"
I20250112 20:51:07.208047 1421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:07.237757 1421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:07.238544 1421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:07.238790 1421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:07.241520 1421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:07.241614 1421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:07.241741 1421 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:07.241791 1421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:07.284150 1421 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:34617
I20250112 20:51:07.284178 1559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:34617 every 8 connection(s)
I20250112 20:51:07.285290 1421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:07.287176 1560 heartbeater.cc:346] Connected to a master server at 127.0.104.62:36025
I20250112 20:51:07.287379 1560 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:07.288004 1560 heartbeater.cc:510] Master 127.0.104.62:36025 requested a full tablet report, sending...
I20250112 20:51:07.289211 1075 ts_manager.cc:194] Registered new tserver with Master: 24a44ead261e4e6aa90d3f612ec86a3b (127.0.104.3:34617)
I20250112 20:51:07.289980 1075 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:37203
I20250112 20:51:07.293606 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 1421
I20250112 20:51:07.293763 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.DeadlockPrevention.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:07.296391 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:07.310631 1075 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52558:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:07.311764 1075 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:07.354966 1216 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1b62eab4d74907ab599a215b8395d8 (DEFAULT_TABLE table=txn_write_ops_test [id=1075e21fc0e644a696f383df64dedf5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:07.355717 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1b62eab4d74907ab599a215b8395d8 (DEFAULT_TABLE table=txn_write_ops_test [id=1075e21fc0e644a696f383df64dedf5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:07.356384 1216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1b62eab4d74907ab599a215b8395d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.354933 1217 tablet_service.cc:1467] Processing CreateTablet for tablet 89bffb1464e248e988bed8086e2202fe (DEFAULT_TABLE table=txn_write_ops_test [id=1075e21fc0e644a696f383df64dedf5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:07.356923 1353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1b62eab4d74907ab599a215b8395d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.354916 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 89bffb1464e248e988bed8086e2202fe (DEFAULT_TABLE table=txn_write_ops_test [id=1075e21fc0e644a696f383df64dedf5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:07.359655 1495 tablet_service.cc:1467] Processing CreateTablet for tablet 89bffb1464e248e988bed8086e2202fe (DEFAULT_TABLE table=txn_write_ops_test [id=1075e21fc0e644a696f383df64dedf5c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:07.360308 1354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bffb1464e248e988bed8086e2202fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.360973 1495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bffb1464e248e988bed8086e2202fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.359714 1494 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1b62eab4d74907ab599a215b8395d8 (DEFAULT_TABLE table=txn_write_ops_test [id=1075e21fc0e644a696f383df64dedf5c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:07.364907 1494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e1b62eab4d74907ab599a215b8395d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.365017 1217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89bffb1464e248e988bed8086e2202fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.374977 1577 tablet_bootstrap.cc:492] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f: Bootstrap starting.
I20250112 20:51:07.380010 1577 tablet_bootstrap.cc:654] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.381368 1577 log.cc:826] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:07.383216 1579 tablet_bootstrap.cc:492] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b: Bootstrap starting.
I20250112 20:51:07.386732 1578 tablet_bootstrap.cc:492] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b: Bootstrap starting.
I20250112 20:51:07.388839 1577 tablet_bootstrap.cc:492] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f: No bootstrap required, opened a new log
I20250112 20:51:07.389158 1577 ts_tablet_manager.cc:1397] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20250112 20:51:07.389880 1578 tablet_bootstrap.cc:654] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.390453 1579 tablet_bootstrap.cc:654] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.391235 1578 log.cc:826] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:07.391641 1579 log.cc:826] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:07.391990 1283 heartbeater.cc:502] Master 127.0.104.62:36025 was elected leader, sending a full tablet report...
I20250112 20:51:07.400259 1577 raft_consensus.cc:357] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.400862 1577 raft_consensus.cc:383] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:07.401021 1577 raft_consensus.cc:738] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a72ff2aef448059e9ebac62bd4646f, State: Initialized, Role: FOLLOWER
I20250112 20:51:07.401269 1579 tablet_bootstrap.cc:492] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b: No bootstrap required, opened a new log
I20250112 20:51:07.401520 1578 tablet_bootstrap.cc:492] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b: No bootstrap required, opened a new log
I20250112 20:51:07.401551 1579 ts_tablet_manager.cc:1397] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.005s
I20250112 20:51:07.401795 1578 ts_tablet_manager.cc:1397] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250112 20:51:07.401971 1577 consensus_queue.cc:260] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.405287 1419 heartbeater.cc:502] Master 127.0.104.62:36025 was elected leader, sending a full tablet report...
I20250112 20:51:07.407227 1577 ts_tablet_manager.cc:1428] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20250112 20:51:07.407629 1577 tablet_bootstrap.cc:492] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f: Bootstrap starting.
I20250112 20:51:07.409104 1579 raft_consensus.cc:357] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.409452 1577 tablet_bootstrap.cc:654] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.409509 1579 raft_consensus.cc:383] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:07.409612 1579 raft_consensus.cc:738] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a44ead261e4e6aa90d3f612ec86a3b, State: Initialized, Role: FOLLOWER
I20250112 20:51:07.410400 1579 consensus_queue.cc:260] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.413341 1560 heartbeater.cc:502] Master 127.0.104.62:36025 was elected leader, sending a full tablet report...
I20250112 20:51:07.413123 1578 raft_consensus.cc:357] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.413668 1578 raft_consensus.cc:383] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:07.413784 1578 raft_consensus.cc:738] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4401b340cbe43c0ab7e465e55d0ce5b, State: Initialized, Role: FOLLOWER
I20250112 20:51:07.414574 1578 consensus_queue.cc:260] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.415176 1579 ts_tablet_manager.cc:1428] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20250112 20:51:07.415594 1579 tablet_bootstrap.cc:492] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b: Bootstrap starting.
I20250112 20:51:07.418082 1579 tablet_bootstrap.cc:654] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.418097 1578 ts_tablet_manager.cc:1428] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20250112 20:51:07.418474 1578 tablet_bootstrap.cc:492] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b: Bootstrap starting.
I20250112 20:51:07.419138 1577 tablet_bootstrap.cc:492] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f: No bootstrap required, opened a new log
I20250112 20:51:07.419291 1577 ts_tablet_manager.cc:1397] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20250112 20:51:07.420179 1578 tablet_bootstrap.cc:654] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.420078 1577 raft_consensus.cc:357] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.420269 1577 raft_consensus.cc:383] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:07.420356 1577 raft_consensus.cc:738] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a72ff2aef448059e9ebac62bd4646f, State: Initialized, Role: FOLLOWER
I20250112 20:51:07.420615 1577 consensus_queue.cc:260] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.421168 1577 ts_tablet_manager.cc:1428] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:07.422571 1578 tablet_bootstrap.cc:492] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b: No bootstrap required, opened a new log
I20250112 20:51:07.422695 1578 ts_tablet_manager.cc:1397] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:07.423112 1579 tablet_bootstrap.cc:492] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b: No bootstrap required, opened a new log
I20250112 20:51:07.423256 1579 ts_tablet_manager.cc:1397] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20250112 20:51:07.423454 1578 raft_consensus.cc:357] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.423640 1578 raft_consensus.cc:383] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:07.423733 1578 raft_consensus.cc:738] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4401b340cbe43c0ab7e465e55d0ce5b, State: Initialized, Role: FOLLOWER
I20250112 20:51:07.423939 1578 consensus_queue.cc:260] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.423949 1579 raft_consensus.cc:357] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.424153 1579 raft_consensus.cc:383] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:07.424278 1579 raft_consensus.cc:738] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a44ead261e4e6aa90d3f612ec86a3b, State: Initialized, Role: FOLLOWER
I20250112 20:51:07.424515 1579 consensus_queue.cc:260] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.424865 1578 ts_tablet_manager.cc:1428] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:07.425014 1579 ts_tablet_manager.cc:1428] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:07.428462 1583 raft_consensus.cc:491] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:07.428686 1583 raft_consensus.cc:513] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.430205 1583 leader_election.cc:290] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145), 24a44ead261e4e6aa90d3f612ec86a3b (127.0.104.3:34617)
I20250112 20:51:07.432979 1585 raft_consensus.cc:491] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:07.433218 1585 raft_consensus.cc:513] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.435474 1585 leader_election.cc:290] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543), a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145)
I20250112 20:51:07.440062 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bffb1464e248e988bed8086e2202fe" candidate_uuid: "52a72ff2aef448059e9ebac62bd4646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" is_pre_election: true
I20250112 20:51:07.440591 1515 raft_consensus.cc:2463] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52a72ff2aef448059e9ebac62bd4646f in term 0.
I20250112 20:51:07.441192 1303 leader_election.cc:304] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a44ead261e4e6aa90d3f612ec86a3b, 52a72ff2aef448059e9ebac62bd4646f; no voters:
I20250112 20:51:07.441557 1583 raft_consensus.cc:2798] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:07.441677 1583 raft_consensus.cc:491] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:07.441817 1583 raft_consensus.cc:3054] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:07.444113 1583 raft_consensus.cc:513] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.444708 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bffb1464e248e988bed8086e2202fe" candidate_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a72ff2aef448059e9ebac62bd4646f" is_pre_election: true
I20250112 20:51:07.445345 1374 raft_consensus.cc:2388] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 24a44ead261e4e6aa90d3f612ec86a3b in current term 1: Already voted for candidate 52a72ff2aef448059e9ebac62bd4646f in this term.
I20250112 20:51:07.445446 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bffb1464e248e988bed8086e2202fe" candidate_uuid: "52a72ff2aef448059e9ebac62bd4646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a44ead261e4e6aa90d3f612ec86a3b"
I20250112 20:51:07.445614 1515 raft_consensus.cc:3054] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:07.444928 1583 leader_election.cc:290] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [CANDIDATE]: Term 1 election: Requested vote from peers a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145), 24a44ead261e4e6aa90d3f612ec86a3b (127.0.104.3:34617)
I20250112 20:51:07.448093 1515 raft_consensus.cc:2463] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52a72ff2aef448059e9ebac62bd4646f in term 1.
I20250112 20:51:07.448451 1303 leader_election.cc:304] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a44ead261e4e6aa90d3f612ec86a3b, 52a72ff2aef448059e9ebac62bd4646f; no voters:
I20250112 20:51:07.448796 1583 raft_consensus.cc:2798] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:07.449427 1583 raft_consensus.cc:695] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [term 1 LEADER]: Becoming Leader. State: Replica: 52a72ff2aef448059e9ebac62bd4646f, State: Running, Role: LEADER
I20250112 20:51:07.450078 1583 consensus_queue.cc:237] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.451882 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bffb1464e248e988bed8086e2202fe" candidate_uuid: "52a72ff2aef448059e9ebac62bd4646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" is_pre_election: true
I20250112 20:51:07.451908 1238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bffb1464e248e988bed8086e2202fe" candidate_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" is_pre_election: true
I20250112 20:51:07.451951 1236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89bffb1464e248e988bed8086e2202fe" candidate_uuid: "52a72ff2aef448059e9ebac62bd4646f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b"
I20250112 20:51:07.452527 1238 raft_consensus.cc:2463] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a44ead261e4e6aa90d3f612ec86a3b in term 0.
I20250112 20:51:07.453159 1442 leader_election.cc:304] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24a44ead261e4e6aa90d3f612ec86a3b, a4401b340cbe43c0ab7e465e55d0ce5b; no voters: 52a72ff2aef448059e9ebac62bd4646f
I20250112 20:51:07.453584 1585 raft_consensus.cc:2758] T 89bffb1464e248e988bed8086e2202fe P 24a44ead261e4e6aa90d3f612ec86a3b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250112 20:51:07.456619 1074 catalog_manager.cc:5526] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f reported cstate change: term changed from 0 to 1, leader changed from <none> to 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "52a72ff2aef448059e9ebac62bd4646f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:07.484766 1586 raft_consensus.cc:491] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:07.484962 1586 raft_consensus.cc:513] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.486523 1586 leader_election.cc:290] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543), 24a44ead261e4e6aa90d3f612ec86a3b (127.0.104.3:34617)
I20250112 20:51:07.493242 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b62eab4d74907ab599a215b8395d8" candidate_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a72ff2aef448059e9ebac62bd4646f" is_pre_election: true
I20250112 20:51:07.493530 1374 raft_consensus.cc:2463] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4401b340cbe43c0ab7e465e55d0ce5b in term 0.
I20250112 20:51:07.493485 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b62eab4d74907ab599a215b8395d8" candidate_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" is_pre_election: true
I20250112 20:51:07.493731 1515 raft_consensus.cc:2463] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4401b340cbe43c0ab7e465e55d0ce5b in term 0.
I20250112 20:51:07.494028 1164 leader_election.cc:304] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a72ff2aef448059e9ebac62bd4646f, a4401b340cbe43c0ab7e465e55d0ce5b; no voters:
I20250112 20:51:07.494552 1586 raft_consensus.cc:2798] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:07.494693 1586 raft_consensus.cc:491] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:07.494791 1586 raft_consensus.cc:3054] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:07.496781 1586 raft_consensus.cc:513] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.497421 1586 leader_election.cc:290] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [CANDIDATE]: Term 1 election: Requested vote from peers 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543), 24a44ead261e4e6aa90d3f612ec86a3b (127.0.104.3:34617)
I20250112 20:51:07.497735 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b62eab4d74907ab599a215b8395d8" candidate_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a72ff2aef448059e9ebac62bd4646f"
I20250112 20:51:07.497898 1374 raft_consensus.cc:3054] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:07.497833 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b62eab4d74907ab599a215b8395d8" candidate_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24a44ead261e4e6aa90d3f612ec86a3b"
I20250112 20:51:07.498025 1515 raft_consensus.cc:3054] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:07.499964 1515 raft_consensus.cc:2463] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4401b340cbe43c0ab7e465e55d0ce5b in term 1.
I20250112 20:51:07.500073 1374 raft_consensus.cc:2463] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4401b340cbe43c0ab7e465e55d0ce5b in term 1.
I20250112 20:51:07.500349 1167 leader_election.cc:304] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a44ead261e4e6aa90d3f612ec86a3b, a4401b340cbe43c0ab7e465e55d0ce5b; no voters:
I20250112 20:51:07.500659 1586 raft_consensus.cc:2798] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:07.501107 1586 raft_consensus.cc:695] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [term 1 LEADER]: Becoming Leader. State: Replica: a4401b340cbe43c0ab7e465e55d0ce5b, State: Running, Role: LEADER
I20250112 20:51:07.501586 1586 consensus_queue.cc:237] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.505963 1074 catalog_manager.cc:5526] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b reported cstate change: term changed from 0 to 1, leader changed from <none> to a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1). New cstate: current_term: 1 leader_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:07.536275 416 test_util.cc:274] Using random seed: 2091725667
W20250112 20:51:07.536829 1561 tablet.cc:2367] T 3e1b62eab4d74907ab599a215b8395d8 P 24a44ead261e4e6aa90d3f612ec86a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:07.538843 1564 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
W20250112 20:51:07.589195 1420 tablet.cc:2367] T 3e1b62eab4d74907ab599a215b8395d8 P 52a72ff2aef448059e9ebac62bd4646f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:07.635877 1284 tablet.cc:2367] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:07.856762 1237 raft_consensus.cc:3054] T 89bffb1464e248e988bed8086e2202fe P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:07.860109 1583 consensus_queue.cc:1035] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:07.865782 1593 consensus_queue.cc:1035] T 89bffb1464e248e988bed8086e2202fe P 52a72ff2aef448059e9ebac62bd4646f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:07.903271 1586 consensus_queue.cc:1035] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:07.908463 1586 consensus_queue.cc:1035] T 3e1b62eab4d74907ab599a215b8395d8 P a4401b340cbe43c0ab7e465e55d0ce5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:07.976145 1075 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:07.976867 1075 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:07.983180 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd561d4f1cd4b23bfe311c93134016e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6988afccf378483598fda942628ec006]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:07.982913 1217 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd561d4f1cd4b23bfe311c93134016e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6988afccf378483598fda942628ec006]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:07.983415 1494 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd561d4f1cd4b23bfe311c93134016e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6988afccf378483598fda942628ec006]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:07.983883 1217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd561d4f1cd4b23bfe311c93134016e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.983884 1353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd561d4f1cd4b23bfe311c93134016e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.984088 1494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fd561d4f1cd4b23bfe311c93134016e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:07.990128 1622 tablet_bootstrap.cc:492] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: Bootstrap starting.
I20250112 20:51:07.990976 1624 tablet_bootstrap.cc:492] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: Bootstrap starting.
I20250112 20:51:07.991856 1622 tablet_bootstrap.cc:654] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.992942 1624 tablet_bootstrap.cc:654] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:07.995318 1624 tablet_bootstrap.cc:492] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: No bootstrap required, opened a new log
I20250112 20:51:07.995491 1624 ts_tablet_manager.cc:1397] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:51:07.996435 1624 raft_consensus.cc:357] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.996840 1624 raft_consensus.cc:383] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:07.996985 1624 raft_consensus.cc:738] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4401b340cbe43c0ab7e465e55d0ce5b, State: Initialized, Role: FOLLOWER
I20250112 20:51:07.997340 1624 consensus_queue.cc:260] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:07.998054 1624 ts_tablet_manager.cc:1428] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:07.997932 1586 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:07.998301 1586 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:07.998395 1623 tablet_bootstrap.cc:492] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: Bootstrap starting.
I20250112 20:51:07.999855 1623 tablet_bootstrap.cc:654] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:08.000416 1622 tablet_bootstrap.cc:492] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: No bootstrap required, opened a new log
I20250112 20:51:08.000586 1622 ts_tablet_manager.cc:1397] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20250112 20:51:08.001345 1622 raft_consensus.cc:357] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:08.001531 1623 tablet_bootstrap.cc:492] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: No bootstrap required, opened a new log
I20250112 20:51:08.001556 1622 raft_consensus.cc:383] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:08.001652 1622 raft_consensus.cc:738] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a72ff2aef448059e9ebac62bd4646f, State: Initialized, Role: FOLLOWER
I20250112 20:51:08.001658 1623 ts_tablet_manager.cc:1397] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:08.001845 1622 consensus_queue.cc:260] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:08.002368 1622 ts_tablet_manager.cc:1428] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:08.002281 1623 raft_consensus.cc:357] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:08.002271 1593 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.002452 1623 raft_consensus.cc:383] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:08.002516 1593 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:08.002543 1623 raft_consensus.cc:738] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a44ead261e4e6aa90d3f612ec86a3b, State: Initialized, Role: FOLLOWER
I20250112 20:51:08.002733 1623 consensus_queue.cc:260] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:08.003216 1623 ts_tablet_manager.cc:1428] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:08.003214 1585 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.003458 1585 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:08.125239 1585 raft_consensus.cc:491] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:08.125430 1585 raft_consensus.cc:513] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:08.126053 1585 leader_election.cc:290] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145), 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543)
I20250112 20:51:08.126631 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd561d4f1cd4b23bfe311c93134016e" candidate_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a72ff2aef448059e9ebac62bd4646f" is_pre_election: true
I20250112 20:51:08.126646 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd561d4f1cd4b23bfe311c93134016e" candidate_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" is_pre_election: true
I20250112 20:51:08.126854 1374 raft_consensus.cc:2463] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a44ead261e4e6aa90d3f612ec86a3b in term 0.
I20250112 20:51:08.126885 1237 raft_consensus.cc:2463] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a44ead261e4e6aa90d3f612ec86a3b in term 0.
I20250112 20:51:08.127270 1441 leader_election.cc:304] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a44ead261e4e6aa90d3f612ec86a3b, 52a72ff2aef448059e9ebac62bd4646f; no voters:
I20250112 20:51:08.127537 1585 raft_consensus.cc:2798] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:08.127653 1585 raft_consensus.cc:491] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:08.127732 1585 raft_consensus.cc:3054] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:08.129680 1585 raft_consensus.cc:513] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:08.130187 1585 leader_election.cc:290] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [CANDIDATE]: Term 1 election: Requested vote from peers a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145), 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543)
I20250112 20:51:08.130684 1237 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd561d4f1cd4b23bfe311c93134016e" candidate_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b"
I20250112 20:51:08.130712 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd561d4f1cd4b23bfe311c93134016e" candidate_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a72ff2aef448059e9ebac62bd4646f"
I20250112 20:51:08.130888 1237 raft_consensus.cc:3054] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:08.130918 1374 raft_consensus.cc:3054] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:08.133670 1374 raft_consensus.cc:2463] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a44ead261e4e6aa90d3f612ec86a3b in term 1.
I20250112 20:51:08.133687 1237 raft_consensus.cc:2463] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a44ead261e4e6aa90d3f612ec86a3b in term 1.
I20250112 20:51:08.134119 1442 leader_election.cc:304] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a44ead261e4e6aa90d3f612ec86a3b, a4401b340cbe43c0ab7e465e55d0ce5b; no voters:
I20250112 20:51:08.134460 1585 raft_consensus.cc:2798] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:08.134820 1585 raft_consensus.cc:695] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [term 1 LEADER]: Becoming Leader. State: Replica: 24a44ead261e4e6aa90d3f612ec86a3b, State: Running, Role: LEADER
I20250112 20:51:08.135264 1585 consensus_queue.cc:237] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } }
I20250112 20:51:08.137171 1628 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: TxnStatusTablet state changed. Reason: New leader 24a44ead261e4e6aa90d3f612ec86a3b. Latest consensus state: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.137403 1628 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:08.138012 1630 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:08.138254 1630 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:08.138808 1075 catalog_manager.cc:5526] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a44ead261e4e6aa90d3f612ec86a3b (127.0.104.3). New cstate: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:08.409296 1237 raft_consensus.cc:1270] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b [term 1 FOLLOWER]: Refusing update from remote peer 24a44ead261e4e6aa90d3f612ec86a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250112 20:51:08.409446 1374 raft_consensus.cc:1270] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f [term 1 FOLLOWER]: Refusing update from remote peer 24a44ead261e4e6aa90d3f612ec86a3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:08.410066 1585 consensus_queue.cc:1035] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:08.410409 1634 consensus_queue.cc:1035] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:08.422307 1597 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: TxnStatusTablet state changed. Reason: New leader 24a44ead261e4e6aa90d3f612ec86a3b. Latest consensus state: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.422562 1597 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:08.425534 1593 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: TxnStatusTablet state changed. Reason: New leader 24a44ead261e4e6aa90d3f612ec86a3b. Latest consensus state: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.425808 1593 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:08.429041 1613 mvcc.cc:204] Tried to move back new op lower bound from 7113584920195284992 to 7113584919085920256. Current Snapshot: MvccSnapshot[applied={T|T < 7113584920195284992}]
I20250112 20:51:08.443053 1597 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.443305 1597 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P a4401b340cbe43c0ab7e465e55d0ce5b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:08.450950 1628 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.451189 1628 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:08.452097 1634 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.452350 1634 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P 24a44ead261e4e6aa90d3f612ec86a3b: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:08.458074 1610 tablet_replica.cc:440] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4401b340cbe43c0ab7e465e55d0ce5b" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46145 } } peers { permanent_uuid: "52a72ff2aef448059e9ebac62bd4646f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34543 } } peers { permanent_uuid: "24a44ead261e4e6aa90d3f612ec86a3b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34617 } } }
I20250112 20:51:08.458437 1610 tablet_replica.cc:443] T 3fd561d4f1cd4b23bfe311c93134016e P 52a72ff2aef448059e9ebac62bd4646f: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:08.544925 1601 txn_write_ops-itest.cc:453] Txn 4 wrote row 3: OK
I20250112 20:51:08.569465 1601 txn_write_ops-itest.cc:453] Txn 4 wrote row 15: OK
I20250112 20:51:08.594058 1602 txn_write_ops-itest.cc:453] Txn 3 wrote row 12: OK
I20250112 20:51:08.603240 1602 txn_write_ops-itest.cc:453] Txn 3 wrote row 1: OK
W20250112 20:51:09.568198 1635 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20250112 20:51:09.568639 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:51:09.569267 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=0, attempt_no=0}) took 1093 ms. Trace:
I20250112 20:51:09.569387 1635 rpcz_store.cc:276] 0112 20:51:08.478849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.478901 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:09.569204 (+1090303us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:08.537956 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:08.538024 (+ 68us) participant_op.cc:199] PREPARE: finished
0112 20:51:08.538047 (+ 23us) participant_op.cc:212] START. Timestamp: 1736715068537996
0112 20:51:08.538049 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:08.538146 (+ 97us) log.cc:844] Serialized 65 byte log entry
0112 20:51:08.564736 (+ 26590us) op_driver.cc:464] REPLICATION: finished
0112 20:51:08.564872 (+ 136us) participant_op.cc:265] APPLY: starting
0112 20:51:08.565089 (+ 217us) participant_op.cc:270] APPLY: finished
0112 20:51:08.565138 (+ 49us) log.cc:844] Serialized 14 byte log entry
0112 20:51:08.565167 (+ 29us) participant_op.cc:94] Released txn lock
0112 20:51:08.565172 (+ 5us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:08.567490 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:08.567583 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:08.567588 (+ 5us) write_op.cc:435] Acquired schema lock
0112 20:51:08.567591 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:08.567657 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:09.568502 (+1000845us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:51:09.568574 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:09.568581 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:09.568589 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":30407,"threads_started":1,"txn-participant-registration.queue_time_us":32568,"txn-participant-registration.run_cpu_time_us":779,"txn-participant-registration.run_wall_time_us":17663,"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":515,"apply.run_wall_time_us":515,"prepare.queue_time_us":1081,"prepare.run_cpu_time_us":264,"prepare.run_wall_time_us":264,"replication_time_us":26657,"spinlock_wait_cycles":83584,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":2135}]]}]]}
W20250112 20:51:09.618609 1666 lock_manager.cc:494] Txn 7 has not acquired the partition lock after 1000ms
W20250112 20:51:09.619194 1666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60442: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:09.619287 1668 ts_tablet_manager.cc:2109] Sending abort request for transaction 7
I20250112 20:51:09.619680 1666 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60442 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=8, attempt_no=0}) took 1015 ms. Trace:
I20250112 20:51:09.619784 1666 rpcz_store.cc:276] 0112 20:51:08.604077 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.604143 (+ 66us) service_pool.cc:225] Handling call
0112 20:51:09.619625 (+1015482us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:08.613472 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:08.613530 (+ 58us) participant_op.cc:199] PREPARE: finished
0112 20:51:08.613557 (+ 27us) participant_op.cc:212] START. Timestamp: 1736715068613502
0112 20:51:08.613560 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:08.613659 (+ 99us) log.cc:844] Serialized 65 byte log entry
0112 20:51:08.616472 (+ 2813us) op_driver.cc:464] REPLICATION: finished
0112 20:51:08.616864 (+ 392us) participant_op.cc:265] APPLY: starting
0112 20:51:08.617090 (+ 226us) participant_op.cc:270] APPLY: finished
0112 20:51:08.617120 (+ 30us) log.cc:844] Serialized 14 byte log entry
0112 20:51:08.617288 (+ 168us) participant_op.cc:94] Released txn lock
0112 20:51:08.617294 (+ 6us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:08.617650 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3e1b62eab4d74907ab599a215b8395d8
0112 20:51:08.617787 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:08.617791 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:51:08.617793 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:08.618090 (+ 297us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:09.618796 (+1000706us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:09.618814 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:09.618821 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:09.618830 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":168,"txn-participant-registration.run_cpu_time_us":635,"txn-participant-registration.run_wall_time_us":8637,"child_traces":[["op",{"apply.queue_time_us":342,"apply.run_cpu_time_us":568,"apply.run_wall_time_us":689,"prepare.queue_time_us":252,"prepare.run_cpu_time_us":278,"prepare.run_wall_time_us":278,"raft.queue_time_us":288,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":456,"replication_time_us":2891,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":135,"thread_start_us":217,"threads_started":1,"txn-participant-registration.queue_time_us":319}]]}]]}
W20250112 20:51:09.621892 1563 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:09.622543 1600 txn_write_ops-itest.cc:453] Txn 7 wrote row 11: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:09.622680 1600 txn_write_ops-itest.cc:464] Txn 7 wrote row 11: Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:51:09.641583 1666 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250112 20:51:09.655221 1679 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:42775 (ReqId={client: ff47fe3768f345da9d0e86206c370c16, seq_no=1, attempt_no=0}) took 1032 ms. Trace:
I20250112 20:51:09.655387 1679 rpcz_store.cc:276] 0112 20:51:08.622841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.622887 (+ 46us) service_pool.cc:225] Handling call
0112 20:51:09.655155 (+1032268us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:09.623360 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:09.623475 (+ 115us) participant_op.cc:199] PREPARE: finished
0112 20:51:09.623537 (+ 62us) participant_op.cc:212] START. Timestamp: 1736715069623451
0112 20:51:09.623539 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:09.623655 (+ 116us) log.cc:844] Serialized 65 byte log entry
0112 20:51:09.654724 (+ 31069us) op_driver.cc:464] REPLICATION: finished
0112 20:51:09.654816 (+ 92us) participant_op.cc:265] APPLY: starting
0112 20:51:09.654959 (+ 143us) participant_op.cc:270] APPLY: finished
0112 20:51:09.655012 (+ 53us) log.cc:844] Serialized 14 byte log entry
0112 20:51:09.655056 (+ 44us) participant_op.cc:94] Released txn lock
0112 20:51:09.655062 (+ 6us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"prepare.queue_time_us":1000134,"prepare.run_cpu_time_us":345,"prepare.run_wall_time_us":605,"replication_time_us":31151}]]}
W20250112 20:51:10.571588 1635 lock_manager.cc:494] Txn 6 has not acquired the partition lock after 1000ms
W20250112 20:51:10.572140 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:10.572265 1690 ts_tablet_manager.cc:2109] Sending abort request for transaction 6
I20250112 20:51:10.572443 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=5, attempt_no=0}) took 2065 ms. Trace:
I20250112 20:51:10.572538 1635 rpcz_store.cc:276] 0112 20:51:08.506698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.506752 (+ 54us) service_pool.cc:225] Handling call
0112 20:51:10.572387 (+2065635us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:08.539776 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:08.539826 (+ 50us) participant_op.cc:199] PREPARE: finished
0112 20:51:08.539849 (+ 23us) participant_op.cc:212] START. Timestamp: 1736715068539798
0112 20:51:08.539854 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 20:51:08.539949 (+ 95us) log.cc:844] Serialized 65 byte log entry
0112 20:51:08.564805 (+ 24856us) op_driver.cc:464] REPLICATION: finished
0112 20:51:08.565441 (+ 636us) participant_op.cc:265] APPLY: starting
0112 20:51:08.565639 (+ 198us) participant_op.cc:270] APPLY: finished
0112 20:51:08.565665 (+ 26us) log.cc:844] Serialized 14 byte log entry
0112 20:51:08.565693 (+ 28us) participant_op.cc:94] Released txn lock
0112 20:51:08.565698 (+ 5us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:09.570856 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:09.570988 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:09.570991 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:51:09.570993 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:09.571086 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:10.571755 (+1000669us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:10.571775 (+ 20us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:10.571784 (+ 9us) write_op.cc:454] Released schema lock
0112 20:51:10.571795 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":13294,"threads_started":1,"txn-participant-registration.queue_time_us":12827,"txn-participant-registration.run_cpu_time_us":655,"txn-participant-registration.run_wall_time_us":18767,"child_traces":[["op",{"apply.queue_time_us":607,"apply.run_cpu_time_us":455,"apply.run_wall_time_us":455,"prepare.queue_time_us":1147,"prepare.run_cpu_time_us":219,"prepare.run_wall_time_us":218,"replication_time_us":24932,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":1004962,"thread_start_us":200,"threads_started":1,"txn-participant-registration.queue_time_us":317}]]}]]}
I20250112 20:51:10.573660 1604 txn_write_ops-itest.cc:453] Txn 6 wrote row 2: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:10.573778 1604 txn_write_ops-itest.cc:464] Txn 6 wrote row 2: Aborted: Failed to write batch of 1 ops to tablet 89bffb1464e248e988bed8086e2202fe after 1 attempt(s): Failed to write to server: 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:51:10.652846 1666 lock_manager.cc:494] Txn 8 has not acquired the partition lock after 1000ms
W20250112 20:51:10.653412 1666 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60442: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:10.653524 1698 ts_tablet_manager.cc:2109] Sending abort request for transaction 8
I20250112 20:51:10.653707 1666 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60442 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=9, attempt_no=0}) took 2045 ms. Trace:
I20250112 20:51:10.653808 1666 rpcz_store.cc:276] 0112 20:51:08.607854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.607905 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:10.653668 (+2045763us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:09.621486 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:09.621554 (+ 68us) participant_op.cc:199] PREPARE: finished
0112 20:51:09.621602 (+ 48us) participant_op.cc:212] START. Timestamp: 1736715069621541
0112 20:51:09.621608 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 20:51:09.621809 (+ 201us) log.cc:844] Serialized 65 byte log entry
0112 20:51:09.651360 (+ 29551us) op_driver.cc:464] REPLICATION: finished
0112 20:51:09.651448 (+ 88us) participant_op.cc:265] APPLY: starting
0112 20:51:09.651805 (+ 357us) participant_op.cc:270] APPLY: finished
0112 20:51:09.651850 (+ 45us) log.cc:844] Serialized 14 byte log entry
0112 20:51:09.651889 (+ 39us) participant_op.cc:94] Released txn lock
0112 20:51:09.651897 (+ 8us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:09.652264 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3e1b62eab4d74907ab599a215b8395d8
0112 20:51:09.652358 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:09.652362 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:51:09.652364 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:09.652442 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:10.653022 (+1000580us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:10.653038 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:10.653047 (+ 9us) write_op.cc:454] Released schema lock
0112 20:51:10.653057 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":645,"threads_started":1,"txn-participant-registration.queue_time_us":802,"txn-participant-registration.run_cpu_time_us":824,"txn-participant-registration.run_wall_time_us":12420,"child_traces":[["op",{"apply.queue_time_us":41,"apply.run_cpu_time_us":757,"apply.run_wall_time_us":756,"prepare.queue_time_us":1000131,"prepare.run_cpu_time_us":718,"prepare.run_wall_time_us":1846,"raft.queue_time_us":184,"raft.run_cpu_time_us":564,"raft.run_wall_time_us":563,"replication_time_us":29720,"thread_start_us":217,"threads_started":1,"wal-append.queue_time_us":296,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":95,"thread_start_us":256,"threads_started":1,"txn-participant-registration.queue_time_us":378}]]}]]}
W20250112 20:51:10.654968 1563 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:10.655303 1605 txn_write_ops-itest.cc:453] Txn 8 wrote row 6: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:10.655391 1605 txn_write_ops-itest.cc:464] Txn 8 wrote row 6: Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:51:10.664809 1666 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
W20250112 20:51:11.574021 1635 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:51:11.574308 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:51:11.574589 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=1, attempt_no=0}) took 3095 ms. Trace:
I20250112 20:51:11.574672 1635 rpcz_store.cc:276] 0112 20:51:08.478990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.481073 (+ 2083us) service_pool.cc:225] Handling call
0112 20:51:11.574531 (+3093458us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:08.558295 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:08.558324 (+ 29us) participant_op.cc:199] PREPARE: finished
0112 20:51:08.558348 (+ 24us) participant_op.cc:212] START. Timestamp: 1736715068558295
0112 20:51:08.558351 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:08.558451 (+ 100us) log.cc:844] Serialized 65 byte log entry
0112 20:51:08.567567 (+ 9116us) op_driver.cc:464] REPLICATION: finished
0112 20:51:08.568333 (+ 766us) participant_op.cc:265] APPLY: starting
0112 20:51:08.568455 (+ 122us) participant_op.cc:270] APPLY: finished
0112 20:51:08.568475 (+ 20us) log.cc:844] Serialized 14 byte log entry
0112 20:51:08.568505 (+ 30us) participant_op.cc:94] Released txn lock
0112 20:51:08.568510 (+ 5us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:10.573348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:10.573492 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:10.573495 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:51:10.573498 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:10.573585 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:11.574256 (+1000671us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:51:11.574271 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:11.574278 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:11.574286 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":33704,"threads_started":1,"txn-participant-registration.queue_time_us":31790,"txn-participant-registration.run_cpu_time_us":965,"txn-participant-registration.run_wall_time_us":36957,"child_traces":[["op",{"apply.queue_time_us":738,"apply.run_cpu_time_us":415,"apply.run_wall_time_us":414,"prepare.queue_time_us":4162,"prepare.run_cpu_time_us":2889,"prepare.run_wall_time_us":8827,"raft.queue_time_us":197,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":9196,"thread_start_us":8285,"threads_started":1,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":2004618}]]}]]}
W20250112 20:51:12.576097 1635 lock_manager.cc:494] Txn 5 has not acquired the partition lock after 1000ms
W20250112 20:51:12.576705 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:12.576807 1708 ts_tablet_manager.cc:2109] Sending abort request for transaction 5
I20250112 20:51:12.576987 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=4, attempt_no=0}) took 4094 ms. Trace:
I20250112 20:51:12.577080 1635 rpcz_store.cc:276] 0112 20:51:08.482727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.482778 (+ 51us) service_pool.cc:225] Handling call
0112 20:51:12.576929 (+4094151us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:08.567159 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:08.567186 (+ 27us) participant_op.cc:199] PREPARE: finished
0112 20:51:08.567206 (+ 20us) participant_op.cc:212] START. Timestamp: 1736715068567153
0112 20:51:08.567208 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:08.567396 (+ 188us) log.cc:844] Serialized 65 byte log entry
0112 20:51:08.571492 (+ 4096us) op_driver.cc:464] REPLICATION: finished
0112 20:51:08.571591 (+ 99us) participant_op.cc:265] APPLY: starting
0112 20:51:08.571932 (+ 341us) participant_op.cc:270] APPLY: finished
0112 20:51:08.571959 (+ 27us) log.cc:844] Serialized 14 byte log entry
0112 20:51:08.571988 (+ 29us) participant_op.cc:94] Released txn lock
0112 20:51:08.571993 (+ 5us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:11.575429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:11.575554 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:11.575556 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:51:11.575557 (+ 1us) tablet.cc:659] Decoding operations
0112 20:51:11.575632 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:12.576250 (+1000618us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:12.576267 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:12.576276 (+ 9us) write_op.cc:454] Released schema lock
0112 20:51:12.576285 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":33483,"threads_started":1,"txn-participant-registration.queue_time_us":32830,"txn-participant-registration.run_cpu_time_us":691,"txn-participant-registration.run_wall_time_us":36151,"child_traces":[["op",{"apply.queue_time_us":53,"apply.run_cpu_time_us":664,"apply.run_wall_time_us":664,"prepare.queue_time_us":12722,"prepare.run_cpu_time_us":295,"prepare.run_wall_time_us":294,"raft.queue_time_us":186,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":165,"replication_time_us":4261,"spinlock_wait_cycles":174848,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":3003188,"thread_start_us":319,"threads_started":1,"txn-participant-registration.queue_time_us":428}]]}]]}
W20250112 20:51:12.578033 1562 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 89bffb1464e248e988bed8086e2202fe after 1 attempt(s): Failed to write to server: 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:12.578238 1607 txn_write_ops-itest.cc:453] Txn 5 wrote row 13: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:12.578315 1607 txn_write_ops-itest.cc:464] Txn 5 wrote row 13: Aborted: Failed to write batch of 1 ops to tablet 89bffb1464e248e988bed8086e2202fe after 1 attempt(s): Failed to write to server: 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:51:12.593757 1717 rpcz_store.cc:267] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:37101 (ReqId={client: ff47fe3768f345da9d0e86206c370c16, seq_no=0, attempt_no=0}) took 3999 ms (client timeout 4996 ms). Trace:
W20250112 20:51:12.593971 1717 rpcz_store.cc:269] 0112 20:51:08.594674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:08.594759 (+ 85us) service_pool.cc:225] Handling call
0112 20:51:12.593655 (+3998896us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:12.577893 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:12.578000 (+ 107us) participant_op.cc:199] PREPARE: finished
0112 20:51:12.578050 (+ 50us) participant_op.cc:212] START. Timestamp: 1736715072578001
0112 20:51:12.578053 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:12.578195 (+ 142us) log.cc:844] Serialized 65 byte log entry
0112 20:51:12.591781 (+ 13586us) op_driver.cc:464] REPLICATION: finished
0112 20:51:12.592719 (+ 938us) participant_op.cc:265] APPLY: starting
0112 20:51:12.592809 (+ 90us) participant_op.cc:270] APPLY: finished
0112 20:51:12.592862 (+ 53us) log.cc:844] Serialized 14 byte log entry
0112 20:51:12.593521 (+ 659us) participant_op.cc:94] Released txn lock
0112 20:51:12.593533 (+ 12us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":873,"prepare.queue_time_us":3982763,"prepare.run_cpu_time_us":13101,"prepare.run_wall_time_us":13784,"raft.queue_time_us":1275,"raft.run_cpu_time_us":491,"raft.run_wall_time_us":1781,"replication_time_us":13693,"thread_start_us":14574,"threads_started":3,"wal-append.queue_time_us":358}]]}
W20250112 20:51:13.592401 1635 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20250112 20:51:13.592640 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:51:13.592880 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=0, attempt_no=1}) took 4020 ms. Trace:
I20250112 20:51:13.592943 1635 rpcz_store.cc:276] 0112 20:51:09.572924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:09.573002 (+ 78us) service_pool.cc:225] Handling call
0112 20:51:13.592834 (+4019832us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:12.591728 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:12.591832 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:12.591835 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:51:12.591839 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:12.591921 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:13.592585 (+1000664us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:51:13.592604 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:13.592610 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:13.592618 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":3018237}]]}
W20250112 20:51:13.593621 1635 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250112 20:51:13.597738 1731 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:37101 (ReqId={client: ff47fe3768f345da9d0e86206c370c16, seq_no=4, attempt_no=0}) took 3017 ms. Trace:
I20250112 20:51:13.597913 1731 rpcz_store.cc:276] 0112 20:51:10.579827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:10.579875 (+ 48us) service_pool.cc:225] Handling call
0112 20:51:13.597391 (+3017516us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:13.593527 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:13.593823 (+ 296us) participant_op.cc:199] PREPARE: finished
0112 20:51:13.593862 (+ 39us) participant_op.cc:212] START. Timestamp: 1736715073593814
0112 20:51:13.593864 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:13.594010 (+ 146us) log.cc:844] Serialized 65 byte log entry
0112 20:51:13.596824 (+ 2814us) op_driver.cc:464] REPLICATION: finished
0112 20:51:13.597140 (+ 316us) participant_op.cc:265] APPLY: starting
0112 20:51:13.597285 (+ 145us) participant_op.cc:270] APPLY: finished
0112 20:51:13.597324 (+ 39us) log.cc:844] Serialized 14 byte log entry
0112 20:51:13.597358 (+ 34us) participant_op.cc:94] Released txn lock
0112 20:51:13.597371 (+ 13us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":258,"prepare.queue_time_us":3013403,"prepare.run_cpu_time_us":1472,"prepare.run_wall_time_us":1471,"raft.queue_time_us":527,"raft.run_cpu_time_us":599,"raft.run_wall_time_us":600,"replication_time_us":2935,"thread_start_us":1070,"threads_started":3,"wal-append.queue_time_us":284}]]}
W20250112 20:51:14.595855 1635 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:51:14.596130 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:51:14.596388 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=1, attempt_no=1}) took 3018 ms. Trace:
I20250112 20:51:14.596477 1635 rpcz_store.cc:276] 0112 20:51:11.578160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:11.578267 (+ 107us) service_pool.cc:225] Handling call
0112 20:51:14.596358 (+3018091us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:13.595041 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:13.595160 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:13.595162 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:51:13.595163 (+ 1us) tablet.cc:659] Decoding operations
0112 20:51:13.595249 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:14.596037 (+1000788us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:51:14.596053 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:14.596062 (+ 9us) write_op.cc:454] Released schema lock
0112 20:51:14.596072 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":2016298}]]}
W20250112 20:51:14.597080 1635 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250112 20:51:14.601464 1746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:37101 (ReqId={client: ff47fe3768f345da9d0e86206c370c16, seq_no=6, attempt_no=0}) took 1968 ms. Trace:
I20250112 20:51:14.601636 1746 rpcz_store.cc:276] 0112 20:51:12.633086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:12.633138 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:14.601403 (+1968265us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:14.597030 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:14.597181 (+ 151us) participant_op.cc:199] PREPARE: finished
0112 20:51:14.597223 (+ 42us) participant_op.cc:212] START. Timestamp: 1736715074597191
0112 20:51:14.597225 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:14.597371 (+ 146us) log.cc:844] Serialized 65 byte log entry
0112 20:51:14.600707 (+ 3336us) op_driver.cc:464] REPLICATION: finished
0112 20:51:14.601139 (+ 432us) participant_op.cc:265] APPLY: starting
0112 20:51:14.601282 (+ 143us) participant_op.cc:270] APPLY: finished
0112 20:51:14.601330 (+ 48us) log.cc:844] Serialized 14 byte log entry
0112 20:51:14.601371 (+ 41us) participant_op.cc:94] Released txn lock
0112 20:51:14.601384 (+ 13us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":371,"prepare.queue_time_us":1963602,"prepare.run_cpu_time_us":755,"prepare.run_wall_time_us":763,"raft.queue_time_us":443,"raft.run_cpu_time_us":679,"raft.run_wall_time_us":680,"replication_time_us":3449,"thread_start_us":605,"threads_started":2}]]}
I20250112 20:51:14.605957 1746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:37101 (ReqId={client: ff47fe3768f345da9d0e86206c370c16, seq_no=7, attempt_no=0}) took 1968 ms. Trace:
I20250112 20:51:14.606076 1746 rpcz_store.cc:276] 0112 20:51:12.637193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:12.637250 (+ 57us) service_pool.cc:225] Handling call
0112 20:51:14.605898 (+1968648us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:14.597831 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:14.597903 (+ 72us) participant_op.cc:199] PREPARE: finished
0112 20:51:14.597919 (+ 16us) participant_op.cc:212] START. Timestamp: 1736715074597895
0112 20:51:14.597921 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:14.598042 (+ 121us) log.cc:844] Serialized 75 byte log entry
0112 20:51:14.605543 (+ 7501us) op_driver.cc:464] REPLICATION: finished
0112 20:51:14.605645 (+ 102us) participant_op.cc:265] APPLY: starting
0112 20:51:14.605786 (+ 141us) participant_op.cc:270] APPLY: finished
0112 20:51:14.605834 (+ 48us) log.cc:844] Serialized 14 byte log entry
0112 20:51:14.605866 (+ 32us) participant_op.cc:94] Released txn lock
0112 20:51:14.605879 (+ 13us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"prepare.queue_time_us":1960353,"prepare.run_cpu_time_us":258,"prepare.run_wall_time_us":257,"replication_time_us":7598,"spinlock_wait_cycles":1280}]]}
I20250112 20:51:14.612056 1746 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=0, attempt_no=2}) took 1012 ms. Trace:
I20250112 20:51:14.612203 1746 rpcz_store.cc:276] 0112 20:51:13.599273 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:13.599348 (+ 75us) service_pool.cc:225] Handling call
0112 20:51:14.611990 (+1012642us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:14.598132 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:14.598261 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:14.598265 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:51:14.598267 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:14.598357 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:14.606223 (+ 7866us) write_op.cc:641] Partition lock acquired for write op
0112 20:51:14.606230 (+ 7us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:14.606289 (+ 59us) tablet.cc:698] Row locks acquired
0112 20:51:14.606292 (+ 3us) write_op.cc:260] PREPARE: finished
0112 20:51:14.606329 (+ 37us) write_op.cc:270] Start()
0112 20:51:14.606356 (+ 27us) write_op.cc:276] Timestamp: P: 1736715074606304 usec, L: 0
0112 20:51:14.606361 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 20:51:14.606511 (+ 150us) log.cc:844] Serialized 293 byte log entry
0112 20:51:14.610875 (+ 4364us) op_driver.cc:464] REPLICATION: finished
0112 20:51:14.611355 (+ 480us) write_op.cc:301] APPLY: starting
0112 20:51:14.611380 (+ 25us) tablet.cc:1364] starting BulkCheckPresence
0112 20:51:14.611448 (+ 68us) tablet.cc:1367] finished BulkCheckPresence
0112 20:51:14.611481 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0112 20:51:14.611584 (+ 103us) tablet.cc:1380] finished ApplyRowOperation cycle
0112 20:51:14.611645 (+ 61us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0112 20:51:14.611656 (+ 11us) write_op.cc:312] APPLY: finished
0112 20:51:14.611738 (+ 82us) log.cc:844] Serialized 24 byte log entry
0112 20:51:14.611855 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:14.611876 (+ 21us) write_op.cc:454] Released schema lock
0112 20:51:14.611884 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":416,"num_ops":1,"prepare.queue_time_us":998271,"prepare.run_cpu_time_us":630,"prepare.run_wall_time_us":8460,"replication_time_us":4482}]]}
I20250112 20:51:14.613821 1603 txn_write_ops-itest.cc:453] Txn 1 wrote row 16: OK
I20250112 20:51:14.659860 1603 txn_write_ops-itest.cc:453] Txn 1 wrote row 9: OK
W20250112 20:51:15.607255 1635 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:51:15.607486 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:51:15.607761 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=1, attempt_no=2}) took 1003 ms. Trace:
I20250112 20:51:15.607833 1635 rpcz_store.cc:276] 0112 20:51:14.604148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:14.604213 (+ 65us) service_pool.cc:225] Handling call
0112 20:51:15.607713 (+1003500us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:14.606634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:14.606725 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:14.606728 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:51:14.606730 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:14.606815 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:15.607413 (+1000598us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:51:15.607443 (+ 30us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:15.607449 (+ 6us) write_op.cc:454] Released schema lock
0112 20:51:15.607457 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":1826}]]}
W20250112 20:51:15.660794 1742 lock_manager.cc:494] Txn 9 has not acquired the partition lock after 1000ms
I20250112 20:51:15.661427 1782 ts_tablet_manager.cc:2109] Sending abort request for transaction 9
W20250112 20:51:15.661479 1742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60442: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:15.661816 1742 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60442 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=11, attempt_no=0}) took 1027 ms. Trace:
I20250112 20:51:15.661940 1742 rpcz_store.cc:276] 0112 20:51:14.634694 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:14.634765 (+ 71us) service_pool.cc:225] Handling call
0112 20:51:15.661741 (+1026976us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:14.654900 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:14.654938 (+ 38us) participant_op.cc:199] PREPARE: finished
0112 20:51:14.654968 (+ 30us) participant_op.cc:212] START. Timestamp: 1736715074654915
0112 20:51:14.654972 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:14.655133 (+ 161us) log.cc:844] Serialized 65 byte log entry
0112 20:51:14.658263 (+ 3130us) op_driver.cc:464] REPLICATION: finished
0112 20:51:14.658894 (+ 631us) participant_op.cc:265] APPLY: starting
0112 20:51:14.659372 (+ 478us) participant_op.cc:270] APPLY: finished
0112 20:51:14.659408 (+ 36us) log.cc:844] Serialized 14 byte log entry
0112 20:51:14.659432 (+ 24us) participant_op.cc:94] Released txn lock
0112 20:51:14.659438 (+ 6us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:14.660154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3e1b62eab4d74907ab599a215b8395d8
0112 20:51:14.660262 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:14.660268 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:14.660272 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:14.660357 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:15.660937 (+1000580us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:15.660949 (+ 12us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:15.660954 (+ 5us) write_op.cc:454] Released schema lock
0112 20:51:15.660960 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":2776,"txn-participant-registration.run_cpu_time_us":573,"txn-participant-registration.run_wall_time_us":13500,"child_traces":[["op",{"apply.queue_time_us":537,"apply.run_cpu_time_us":876,"apply.run_wall_time_us":876,"prepare.queue_time_us":3432,"prepare.run_cpu_time_us":295,"prepare.run_wall_time_us":295,"replication_time_us":3267,"spinlock_wait_cycles":74496,"thread_start_us":1700,"threads_started":1,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":414,"thread_start_us":432,"threads_started":1,"txn-participant-registration.queue_time_us":358}]]}]]}
W20250112 20:51:15.663020 1563 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:15.663211 1600 txn_write_ops-itest.cc:453] Txn 9 wrote row 11: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:15.663322 1600 txn_write_ops-itest.cc:464] Txn 9 wrote row 11: Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:51:15.677884 1742 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
W20250112 20:51:16.613801 1635 lock_manager.cc:494] Txn 10 has not acquired the partition lock after 1000ms
W20250112 20:51:16.614279 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:16.614408 1791 ts_tablet_manager.cc:2109] Sending abort request for transaction 10
I20250112 20:51:16.614615 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=12, attempt_no=0}) took 1031 ms. Trace:
I20250112 20:51:16.614701 1635 rpcz_store.cc:276] 0112 20:51:15.582769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:15.582833 (+ 64us) service_pool.cc:225] Handling call
0112 20:51:16.614543 (+1031710us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:15.609040 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:15.609072 (+ 32us) participant_op.cc:199] PREPARE: finished
0112 20:51:15.609099 (+ 27us) participant_op.cc:212] START. Timestamp: 1736715075609055
0112 20:51:15.609103 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:15.609231 (+ 128us) log.cc:844] Serialized 65 byte log entry
0112 20:51:15.611383 (+ 2152us) op_driver.cc:464] REPLICATION: finished
0112 20:51:15.611892 (+ 509us) participant_op.cc:265] APPLY: starting
0112 20:51:15.612207 (+ 315us) participant_op.cc:270] APPLY: finished
0112 20:51:15.612238 (+ 31us) log.cc:844] Serialized 14 byte log entry
0112 20:51:15.612268 (+ 30us) participant_op.cc:94] Released txn lock
0112 20:51:15.612284 (+ 16us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:15.613124 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:15.613271 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:15.613275 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:51:15.613277 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:15.613365 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:16.613970 (+1000605us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:16.613995 (+ 25us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:16.614003 (+ 8us) write_op.cc:454] Released schema lock
0112 20:51:16.614011 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":198,"threads_started":1,"txn-participant-registration.queue_time_us":292,"txn-participant-registration.run_cpu_time_us":647,"txn-participant-registration.run_wall_time_us":5032,"child_traces":[["op",{"apply.queue_time_us":446,"apply.run_cpu_time_us":712,"apply.run_wall_time_us":1388,"prepare.queue_time_us":20650,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":237,"replication_time_us":2255,"spinlock_wait_cycles":24320,"thread_start_us":323,"threads_started":1,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":558,"thread_start_us":179,"threads_started":1,"txn-participant-registration.queue_time_us":306}]]}]]}
I20250112 20:51:16.615864 1604 txn_write_ops-itest.cc:453] Txn 10 wrote row 2: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:16.615993 1604 txn_write_ops-itest.cc:464] Txn 10 wrote row 2: Aborted: Failed to write batch of 1 ops to tablet 89bffb1464e248e988bed8086e2202fe after 1 attempt(s): Failed to write to server: 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:51:16.689083 1742 lock_manager.cc:494] Txn 11 has not acquired the partition lock after 1000ms
W20250112 20:51:16.689594 1742 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60442: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:16.689698 1799 ts_tablet_manager.cc:2109] Sending abort request for transaction 11
I20250112 20:51:16.689867 1742 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60442 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=13, attempt_no=0}) took 1024 ms. Trace:
I20250112 20:51:16.689970 1742 rpcz_store.cc:276] 0112 20:51:15.665510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:15.665762 (+ 252us) service_pool.cc:225] Handling call
0112 20:51:16.689826 (+1024064us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:15.678585 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:15.678615 (+ 30us) participant_op.cc:199] PREPARE: finished
0112 20:51:15.678640 (+ 25us) participant_op.cc:212] START. Timestamp: 1736715075678601
0112 20:51:15.678642 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:15.678745 (+ 103us) log.cc:844] Serialized 65 byte log entry
0112 20:51:15.681952 (+ 3207us) op_driver.cc:464] REPLICATION: finished
0112 20:51:15.682162 (+ 210us) participant_op.cc:265] APPLY: starting
0112 20:51:15.682379 (+ 217us) participant_op.cc:270] APPLY: finished
0112 20:51:15.682417 (+ 38us) log.cc:844] Serialized 14 byte log entry
0112 20:51:15.682438 (+ 21us) participant_op.cc:94] Released txn lock
0112 20:51:15.682444 (+ 6us) participant_op.cc:291] FINISH: Op applied
0112 20:51:15.688633 (+ 6189us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x60d0000c7310. stack: 00007fb365d16fd6 00007fb365d16d90 00007fb364cf648d 000000000096d544 000000000096d50d 000000000096ceda 00007fb3728c3b70 00007fb3728c82df 00007fb3728c7f96 00007fb3790c833a 00007fb3728dacc4 00007fb37291f106 00007fb37291d3a8 00007fb372920420 00007fb37292022d 0000000000a71ecc
Related trace 'op':
0112 20:51:15.688382 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3e1b62eab4d74907ab599a215b8395d8
0112 20:51:15.688492 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:15.688497 (+ 5us) write_op.cc:435] Acquired schema lock
0112 20:51:15.688501 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:15.688602 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:16.689249 (+1000647us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:16.689261 (+ 12us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:16.689268 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:16.689279 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":21939,"threads_started":1,"txn-participant-registration.queue_time_us":5762,"txn-participant-registration.run_cpu_time_us":691,"txn-participant-registration.run_wall_time_us":6714,"child_traces":[["op",{"apply.queue_time_us":142,"apply.run_cpu_time_us":1054,"apply.run_wall_time_us":6538,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":219,"prepare.run_wall_time_us":219,"raft.queue_time_us":320,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":3287,"spinlock_wait_cycles":12932864,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":41,"thread_start_us":197,"threads_started":1,"txn-participant-registration.queue_time_us":327}]]}]]}
W20250112 20:51:16.691056 1563 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:16.691397 1605 txn_write_ops-itest.cc:453] Txn 11 wrote row 6: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:16.691484 1605 txn_write_ops-itest.cc:464] Txn 11 wrote row 6: Aborted: Failed to write batch of 1 ops to tablet 3e1b62eab4d74907ab599a215b8395d8 after 1 attempt(s): Failed to write to server: a4401b340cbe43c0ab7e465e55d0ce5b (127.0.104.1:46145): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:16.715718 1798 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:42775 (ReqId={client: ff47fe3768f345da9d0e86206c370c16, seq_no=11, attempt_no=0}) took 1026 ms. Trace:
I20250112 20:51:16.715881 1798 rpcz_store.cc:276] 0112 20:51:15.688920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:15.689013 (+ 93us) service_pool.cc:225] Handling call
0112 20:51:16.715632 (+1026619us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:16.690864 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:16.690912 (+ 48us) participant_op.cc:199] PREPARE: finished
0112 20:51:16.690929 (+ 17us) participant_op.cc:212] START. Timestamp: 1736715076690893
0112 20:51:16.690931 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:16.691066 (+ 135us) log.cc:844] Serialized 75 byte log entry
0112 20:51:16.710874 (+ 19808us) op_driver.cc:464] REPLICATION: finished
0112 20:51:16.715283 (+ 4409us) participant_op.cc:265] APPLY: starting
0112 20:51:16.715493 (+ 210us) participant_op.cc:270] APPLY: finished
0112 20:51:16.715545 (+ 52us) log.cc:844] Serialized 14 byte log entry
0112 20:51:16.715588 (+ 43us) participant_op.cc:94] Released txn lock
0112 20:51:16.715606 (+ 18us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4350,"prepare.queue_time_us":1001665,"prepare.run_cpu_time_us":1527,"prepare.run_wall_time_us":1527,"raft.queue_time_us":575,"raft.run_cpu_time_us":575,"raft.run_wall_time_us":710,"replication_time_us":19903,"thread_start_us":1193,"threads_started":3,"wal-append.queue_time_us":627}]]}
W20250112 20:51:16.718087 1742 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
W20250112 20:51:17.616232 1635 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:51:17.616466 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:51:17.616777 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=1, attempt_no=3}) took 2002 ms. Trace:
I20250112 20:51:17.616855 1635 rpcz_store.cc:276] 0112 20:51:15.614104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:15.614175 (+ 71us) service_pool.cc:225] Handling call
0112 20:51:17.616722 (+2002547us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:16.615553 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:16.615671 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:16.615676 (+ 5us) write_op.cc:435] Acquired schema lock
0112 20:51:16.615679 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:16.615777 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:17.616399 (+1000622us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:51:17.616414 (+ 15us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:17.616425 (+ 11us) write_op.cc:454] Released schema lock
0112 20:51:17.616435 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":1000973}]]}
W20250112 20:51:17.618763 1635 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250112 20:51:17.621709 1817 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.ParticipateInTransaction from 127.0.104.3:37101 (ReqId={client: ff47fe3768f345da9d0e86206c370c16, seq_no=12, attempt_no=0}) took 1935 ms. Trace:
I20250112 20:51:17.621822 1817 rpcz_store.cc:276] 0112 20:51:15.686272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:15.686319 (+ 47us) service_pool.cc:225] Handling call
0112 20:51:17.621639 (+1935320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:17.617424 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:17.617471 (+ 47us) participant_op.cc:199] PREPARE: finished
0112 20:51:17.617487 (+ 16us) participant_op.cc:212] START. Timestamp: 1736715077617455
0112 20:51:17.617489 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:51:17.617631 (+ 142us) log.cc:844] Serialized 75 byte log entry
0112 20:51:17.621228 (+ 3597us) op_driver.cc:464] REPLICATION: finished
0112 20:51:17.621366 (+ 138us) participant_op.cc:265] APPLY: starting
0112 20:51:17.621517 (+ 151us) participant_op.cc:270] APPLY: finished
0112 20:51:17.621561 (+ 44us) log.cc:844] Serialized 14 byte log entry
0112 20:51:17.621603 (+ 42us) participant_op.cc:94] Released txn lock
0112 20:51:17.621618 (+ 15us) participant_op.cc:291] FINISH: Op applied
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42,"prepare.queue_time_us":1930838,"prepare.run_cpu_time_us":1216,"prepare.run_wall_time_us":1216,"raft.queue_time_us":634,"raft.run_cpu_time_us":686,"raft.run_wall_time_us":688,"replication_time_us":3649,"spinlock_wait_cycles":87424,"thread_start_us":880,"threads_started":2,"wal-append.queue_time_us":346}]]}
I20250112 20:51:17.642205 1606 txn_write_ops-itest.cc:453] Txn 0 wrote row 8: OK
I20250112 20:51:17.657143 1606 txn_write_ops-itest.cc:453] Txn 0 wrote row 5: OK
W20250112 20:51:18.626642 1635 lock_manager.cc:494] Txn 12 has not acquired the partition lock after 1000ms
W20250112 20:51:18.627177 1635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48650: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:18.627295 1835 ts_tablet_manager.cc:2109] Sending abort request for transaction 12
I20250112 20:51:18.627468 1635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48650 (ReqId={client: a64564ecf7cc4a4fa9c71ff6c5a2ee9d, seq_no=14, attempt_no=0}) took 1039 ms. Trace:
I20250112 20:51:18.627591 1635 rpcz_store.cc:276] 0112 20:51:17.587870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:17.587943 (+ 73us) service_pool.cc:225] Handling call
0112 20:51:18.627422 (+1039479us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:17.619255 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:51:17.619292 (+ 37us) participant_op.cc:199] PREPARE: finished
0112 20:51:17.619318 (+ 26us) participant_op.cc:212] START. Timestamp: 1736715077619281
0112 20:51:17.619321 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:17.619432 (+ 111us) log.cc:844] Serialized 65 byte log entry
0112 20:51:17.624482 (+ 5050us) op_driver.cc:464] REPLICATION: finished
0112 20:51:17.625221 (+ 739us) participant_op.cc:265] APPLY: starting
0112 20:51:17.625462 (+ 241us) participant_op.cc:270] APPLY: finished
0112 20:51:17.625493 (+ 31us) log.cc:844] Serialized 14 byte log entry
0112 20:51:17.625516 (+ 23us) participant_op.cc:94] Released txn lock
0112 20:51:17.625521 (+ 5us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:51:17.626006 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 89bffb1464e248e988bed8086e2202fe
0112 20:51:17.626102 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:17.626105 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:51:17.626108 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:17.626166 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:51:18.626802 (+1000636us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:51:18.626815 (+ 13us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:18.626822 (+ 7us) write_op.cc:454] Released schema lock
0112 20:51:18.626830 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":227,"threads_started":1,"txn-participant-registration.queue_time_us":369,"txn-participant-registration.run_cpu_time_us":647,"txn-participant-registration.run_wall_time_us":6119,"child_traces":[["op",{"apply.queue_time_us":705,"apply.run_cpu_time_us":537,"apply.run_wall_time_us":536,"prepare.queue_time_us":24535,"prepare.run_cpu_time_us":254,"prepare.run_wall_time_us":253,"replication_time_us":5147,"spinlock_wait_cycles":75008,"thread_start_us":12928,"threads_started":1,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":281,"thread_start_us":233,"threads_started":1,"txn-participant-registration.queue_time_us":362}]]}]]}
W20250112 20:51:18.628522 1562 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 89bffb1464e248e988bed8086e2202fe after 1 attempt(s): Failed to write to server: 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:51:18.628796 1607 txn_write_ops-itest.cc:453] Txn 12 wrote row 13: IO error: failed to flush data: error details are available via KuduSession::GetPendingErrors()
I20250112 20:51:18.628901 1607 txn_write_ops-itest.cc:464] Txn 12 wrote row 13: Aborted: Failed to write batch of 1 ops to tablet 89bffb1464e248e988bed8086e2202fe after 1 attempt(s): Failed to write to server: 52a72ff2aef448059e9ebac62bd4646f (127.0.104.2:34543): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:51:18.639353 1635 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250112 20:51:20.688580 1600 txn_write_ops-itest.cc:453] Txn 13 wrote row 11: OK
I20250112 20:51:20.708122 1600 txn_write_ops-itest.cc:453] Txn 13 wrote row 7: OK
I20250112 20:51:21.638741 1604 txn_write_ops-itest.cc:453] Txn 14 wrote row 2: OK
I20250112 20:51:21.652722 1604 txn_write_ops-itest.cc:453] Txn 14 wrote row 14: OK
I20250112 20:51:21.711251 1605 txn_write_ops-itest.cc:453] Txn 15 wrote row 6: OK
I20250112 20:51:21.716722 1605 txn_write_ops-itest.cc:453] Txn 15 wrote row 10: OK
I20250112 20:51:23.656385 1607 txn_write_ops-itest.cc:453] Txn 16 wrote row 13: OK
I20250112 20:51:23.671603 1607 txn_write_ops-itest.cc:453] Txn 16 wrote row 4: OK
I20250112 20:51:23.730561 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 1149
I20250112 20:51:23.731667 1278 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:23.893705 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 1149
I20250112 20:51:23.914399 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 1285
I20250112 20:51:23.915417 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:24.039104 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 1285
I20250112 20:51:24.059962 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 1421
I20250112 20:51:24.061020 1555 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:24.187305 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 1421
I20250112 20:51:24.209733 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 1044
I20250112 20:51:24.210829 1115 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:24.322259 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 1044
2025-01-12T20:51:24Z chronyd exiting
[ OK ] TxnWriteOpsITest.DeadlockPrevention (18988 ms)
[ RUN ] TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds
2025-01-12T20:51:24Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:24Z Disabled control of system clock
I20250112 20:51:24.389482 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:45501
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:45501
--txn_manager_enabled=true
--txn_manager_status_table_range_partition_span=3 with env {}
W20250112 20:51:24.706609 1936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:24.707065 1936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:24.707223 1936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:24.717604 1936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:24.717713 1936 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:24.717757 1936 flags.cc:425] Enabled experimental flag: --txn_manager_status_table_range_partition_span=3
W20250112 20:51:24.717831 1936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:24.717892 1936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:24.717972 1936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:24.727213 1936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:45501
--txn_manager_enabled=true
--txn_manager_status_table_range_partition_span=3
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:45501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:24.728821 1936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:24.730472 1936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:24.740532 1944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:24.740581 1941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:24.740612 1942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:24.741376 1936 server_base.cc:1034] running on GCE node
I20250112 20:51:24.742120 1936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:24.743525 1936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:24.744762 1936 hybrid_clock.cc:648] HybridClock initialized: now 1736715084744700 us; error 43 us; skew 500 ppm
I20250112 20:51:24.745090 1936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:24.747330 1936 webserver.cc:458] Webserver started at http://127.0.104.62:33549/ using document root <none> and password file <none>
I20250112 20:51:24.748050 1936 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:24.748168 1936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:24.748466 1936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:24.751075 1936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "7a11a4cd17aa445983ef5ae082e2e413"
format_stamp: "Formatted at 2025-01-12 20:51:24 on dist-test-slave-3gql"
I20250112 20:51:24.751820 1936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "7a11a4cd17aa445983ef5ae082e2e413"
format_stamp: "Formatted at 2025-01-12 20:51:24 on dist-test-slave-3gql"
I20250112 20:51:24.756978 1936 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250112 20:51:24.760113 1950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:24.761396 1936 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:24.761545 1936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "7a11a4cd17aa445983ef5ae082e2e413"
format_stamp: "Formatted at 2025-01-12 20:51:24 on dist-test-slave-3gql"
I20250112 20:51:24.761727 1936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:24.776538 1936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:24.777377 1936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:24.777663 1936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:24.797580 1936 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:45501
I20250112 20:51:24.797624 2011 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:45501 every 8 connection(s)
I20250112 20:51:24.798905 1936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:24.802656 2012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:24.805634 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 1936
I20250112 20:51:24.805840 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:24.813131 2012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: Bootstrap starting.
I20250112 20:51:24.816933 2012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:24.818307 2012 log.cc:826] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:24.821002 2012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: No bootstrap required, opened a new log
I20250112 20:51:24.826946 2012 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a11a4cd17aa445983ef5ae082e2e413" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45501 } }
I20250112 20:51:24.827287 2012 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:24.827379 2012 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a11a4cd17aa445983ef5ae082e2e413, State: Initialized, Role: FOLLOWER
I20250112 20:51:24.827950 2012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a11a4cd17aa445983ef5ae082e2e413" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45501 } }
I20250112 20:51:24.828178 2012 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:24.828330 2012 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:24.828588 2012 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:24.830595 2012 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a11a4cd17aa445983ef5ae082e2e413" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45501 } }
I20250112 20:51:24.831005 2012 leader_election.cc:304] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a11a4cd17aa445983ef5ae082e2e413; no voters:
I20250112 20:51:24.831534 2012 leader_election.cc:290] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:24.831657 2017 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:24.832228 2017 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [term 1 LEADER]: Becoming Leader. State: Replica: 7a11a4cd17aa445983ef5ae082e2e413, State: Running, Role: LEADER
I20250112 20:51:24.832796 2017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a11a4cd17aa445983ef5ae082e2e413" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45501 } }
I20250112 20:51:24.833400 2012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:24.835289 2018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a11a4cd17aa445983ef5ae082e2e413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a11a4cd17aa445983ef5ae082e2e413" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45501 } } }
I20250112 20:51:24.836035 2018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:24.835677 2019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a11a4cd17aa445983ef5ae082e2e413. Latest consensus state: current_term: 1 leader_uuid: "7a11a4cd17aa445983ef5ae082e2e413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a11a4cd17aa445983ef5ae082e2e413" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45501 } } }
I20250112 20:51:24.836654 2019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:24.838023 2024 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:24.843511 2024 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:24.852012 2024 catalog_manager.cc:1348] Generated new cluster ID: c615ef1bdd47461999e329833f5b5f61
I20250112 20:51:24.852142 2024 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:24.879333 2024 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:24.880959 2024 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:24.889552 2024 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: Generated new TSK 0
I20250112 20:51:24.890388 2024 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:24.895530 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45501
--builtin_ntp_servers=127.0.104.20:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:25.213344 2036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:25.213650 2036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:25.213788 2036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:25.221184 2036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:25.221357 2036 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:25.221495 2036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:25.230060 2036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:45501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:25.231650 2036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:25.233246 2036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:25.243700 2041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:25.243779 2042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:25.243739 2044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:25.244791 2036 server_base.cc:1034] running on GCE node
I20250112 20:51:25.245357 2036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:25.246232 2036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:25.247442 2036 hybrid_clock.cc:648] HybridClock initialized: now 1736715085247388 us; error 40 us; skew 500 ppm
I20250112 20:51:25.247725 2036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:25.249890 2036 webserver.cc:458] Webserver started at http://127.0.104.1:41575/ using document root <none> and password file <none>
I20250112 20:51:25.250662 2036 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:25.250751 2036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:25.251056 2036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:25.253722 2036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "23aa8791e86440869c09782df3a8d6c9"
format_stamp: "Formatted at 2025-01-12 20:51:25 on dist-test-slave-3gql"
I20250112 20:51:25.254382 2036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "23aa8791e86440869c09782df3a8d6c9"
format_stamp: "Formatted at 2025-01-12 20:51:25 on dist-test-slave-3gql"
I20250112 20:51:25.259187 2036 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:25.262173 2050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:25.263358 2036 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:25.263499 2036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "23aa8791e86440869c09782df3a8d6c9"
format_stamp: "Formatted at 2025-01-12 20:51:25 on dist-test-slave-3gql"
I20250112 20:51:25.263648 2036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:25.277105 2036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:25.277833 2036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:25.278072 2036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:25.280685 2036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:25.280836 2036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:25.280983 2036 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:25.281055 2036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:25.322085 2036 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:38751
I20250112 20:51:25.322117 2169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:38751 every 8 connection(s)
I20250112 20:51:25.323414 2036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:25.323706 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 2036
I20250112 20:51:25.323894 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:25.325572 2170 heartbeater.cc:346] Connected to a master server at 127.0.104.62:45501
I20250112 20:51:25.325762 2170 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:25.326407 2170 heartbeater.cc:510] Master 127.0.104.62:45501 requested a full tablet report, sending...
I20250112 20:51:25.327546 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45501
--builtin_ntp_servers=127.0.104.20:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:25.328358 1966 ts_manager.cc:194] Registered new tserver with Master: 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751)
I20250112 20:51:25.329908 1966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:43753
W20250112 20:51:25.641285 2172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:25.641573 2172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:25.641711 2172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:25.649323 2172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:25.649503 2172 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:25.649734 2172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:25.659283 2172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:45501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:25.660765 2172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:25.662386 2172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:25.673154 2180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:25.673190 2178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:25.673121 2177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:25.674258 2172 server_base.cc:1034] running on GCE node
I20250112 20:51:25.674822 2172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:25.675691 2172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:25.676939 2172 hybrid_clock.cc:648] HybridClock initialized: now 1736715085676857 us; error 69 us; skew 500 ppm
I20250112 20:51:25.677250 2172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:25.679476 2172 webserver.cc:458] Webserver started at http://127.0.104.2:39179/ using document root <none> and password file <none>
I20250112 20:51:25.680191 2172 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:25.680299 2172 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:25.680656 2172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:25.683310 2172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
format_stamp: "Formatted at 2025-01-12 20:51:25 on dist-test-slave-3gql"
I20250112 20:51:25.684065 2172 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
format_stamp: "Formatted at 2025-01-12 20:51:25 on dist-test-slave-3gql"
I20250112 20:51:25.689134 2172 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20250112 20:51:25.692379 2186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:25.693704 2172 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:25.693868 2172 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
format_stamp: "Formatted at 2025-01-12 20:51:25 on dist-test-slave-3gql"
I20250112 20:51:25.694097 2172 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:25.722343 2172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:25.723179 2172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:25.723454 2172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:25.726280 2172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:25.726370 2172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:25.726507 2172 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:25.726608 2172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:25.766232 2172 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41097
I20250112 20:51:25.766299 2305 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41097 every 8 connection(s)
I20250112 20:51:25.767490 2172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:25.769645 2306 heartbeater.cc:346] Connected to a master server at 127.0.104.62:45501
I20250112 20:51:25.769893 2306 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:25.770695 2306 heartbeater.cc:510] Master 127.0.104.62:45501 requested a full tablet report, sending...
I20250112 20:51:25.771988 1966 ts_manager.cc:194] Registered new tserver with Master: 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097)
I20250112 20:51:25.772804 1966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:39765
I20250112 20:51:25.773857 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 2172
I20250112 20:51:25.774055 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:25.777120 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45501
--builtin_ntp_servers=127.0.104.20:40311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:26.095292 2308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:26.095585 2308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:26.095721 2308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:26.103425 2308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:26.103613 2308 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:26.103754 2308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:26.112664 2308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:45501
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:26.114197 2308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:26.115870 2308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:26.126772 2313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:26.126860 2316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:26.126863 2314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:26.127588 2308 server_base.cc:1034] running on GCE node
I20250112 20:51:26.128175 2308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:26.129055 2308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:26.130252 2308 hybrid_clock.cc:648] HybridClock initialized: now 1736715086130207 us; error 36 us; skew 500 ppm
I20250112 20:51:26.130586 2308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:26.132871 2308 webserver.cc:458] Webserver started at http://127.0.104.3:40439/ using document root <none> and password file <none>
I20250112 20:51:26.133641 2308 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:26.133749 2308 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:26.134070 2308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:26.136852 2308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "d45e7b3cee7541df91a1450d5d8054fa"
format_stamp: "Formatted at 2025-01-12 20:51:26 on dist-test-slave-3gql"
I20250112 20:51:26.137691 2308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "d45e7b3cee7541df91a1450d5d8054fa"
format_stamp: "Formatted at 2025-01-12 20:51:26 on dist-test-slave-3gql"
I20250112 20:51:26.142668 2308 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20250112 20:51:26.145622 2322 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:26.146889 2308 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:51:26.147023 2308 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "d45e7b3cee7541df91a1450d5d8054fa"
format_stamp: "Formatted at 2025-01-12 20:51:26 on dist-test-slave-3gql"
I20250112 20:51:26.147208 2308 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:26.166251 2308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:26.167088 2308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:26.167356 2308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:26.170142 2308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:26.170239 2308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:26.170395 2308 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:26.170454 2308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:26.210697 2308 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:41043
I20250112 20:51:26.210731 2441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:41043 every 8 connection(s)
I20250112 20:51:26.211894 2308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:26.213788 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 2308
I20250112 20:51:26.213997 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:26.214006 2442 heartbeater.cc:346] Connected to a master server at 127.0.104.62:45501
I20250112 20:51:26.214248 2442 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:26.215086 2442 heartbeater.cc:510] Master 127.0.104.62:45501 requested a full tablet report, sending...
I20250112 20:51:26.216378 1966 ts_manager.cc:194] Registered new tserver with Master: d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3:41043)
I20250112 20:51:26.217334 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:26.217331 1966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:34255
I20250112 20:51:26.231519 1966 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47566:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:26.237186 1966 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.282773 2241 tablet_service.cc:1467] Processing CreateTablet for tablet 2b58592de9fb45749b79a21c10a7ca65 (DEFAULT_TABLE table=txn_write_ops_test [id=d5365175172b48c2b7dd994ce49f5b54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.282725 2371 tablet_service.cc:1467] Processing CreateTablet for tablet 231f016eec6148bd8145f2ce5eab971c (DEFAULT_TABLE table=txn_write_ops_test [id=d5365175172b48c2b7dd994ce49f5b54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.283540 2240 tablet_service.cc:1467] Processing CreateTablet for tablet 231f016eec6148bd8145f2ce5eab971c (DEFAULT_TABLE table=txn_write_ops_test [id=d5365175172b48c2b7dd994ce49f5b54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.282692 2372 tablet_service.cc:1467] Processing CreateTablet for tablet 2b58592de9fb45749b79a21c10a7ca65 (DEFAULT_TABLE table=txn_write_ops_test [id=d5365175172b48c2b7dd994ce49f5b54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.284329 2371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 231f016eec6148bd8145f2ce5eab971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.284399 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 231f016eec6148bd8145f2ce5eab971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.285377 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 231f016eec6148bd8145f2ce5eab971c (DEFAULT_TABLE table=txn_write_ops_test [id=d5365175172b48c2b7dd994ce49f5b54]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.285511 2104 tablet_service.cc:1467] Processing CreateTablet for tablet 2b58592de9fb45749b79a21c10a7ca65 (DEFAULT_TABLE table=txn_write_ops_test [id=d5365175172b48c2b7dd994ce49f5b54]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:26.286533 2241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b58592de9fb45749b79a21c10a7ca65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.286779 2104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b58592de9fb45749b79a21c10a7ca65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.287751 2103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 231f016eec6148bd8145f2ce5eab971c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.288215 2372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b58592de9fb45749b79a21c10a7ca65. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.306470 2459 tablet_bootstrap.cc:492] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Bootstrap starting.
I20250112 20:51:26.306470 2460 tablet_bootstrap.cc:492] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9: Bootstrap starting.
I20250112 20:51:26.309391 2461 tablet_bootstrap.cc:492] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa: Bootstrap starting.
I20250112 20:51:26.310142 2460 tablet_bootstrap.cc:654] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.310144 2459 tablet_bootstrap.cc:654] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.311435 2459 log.cc:826] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:26.311443 2460 log.cc:826] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:26.312017 2461 tablet_bootstrap.cc:654] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.313560 2461 log.cc:826] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:26.314317 2459 tablet_bootstrap.cc:492] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a: No bootstrap required, opened a new log
I20250112 20:51:26.314342 2460 tablet_bootstrap.cc:492] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9: No bootstrap required, opened a new log
I20250112 20:51:26.314568 2460 ts_tablet_manager.cc:1397] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250112 20:51:26.314590 2459 ts_tablet_manager.cc:1397] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250112 20:51:26.316277 2461 tablet_bootstrap.cc:492] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa: No bootstrap required, opened a new log
I20250112 20:51:26.316536 2461 ts_tablet_manager.cc:1397] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250112 20:51:26.320379 2459 raft_consensus.cc:357] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.320768 2459 raft_consensus.cc:383] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.320904 2459 raft_consensus.cc:738] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7299d0bd6a4d9f8f7212ada1f7da3a, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.321619 2459 consensus_queue.cc:260] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.323377 2460 raft_consensus.cc:357] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.323810 2460 raft_consensus.cc:383] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.323930 2460 raft_consensus.cc:738] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.324564 2460 consensus_queue.cc:260] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.326474 2461 raft_consensus.cc:357] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.326979 2461 raft_consensus.cc:383] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.327108 2461 raft_consensus.cc:738] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.327996 2461 consensus_queue.cc:260] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.330693 2459 ts_tablet_manager.cc:1428] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent starting tablet: real 0.016s user 0.004s sys 0.013s
I20250112 20:51:26.330788 2306 heartbeater.cc:502] Master 127.0.104.62:45501 was elected leader, sending a full tablet report...
I20250112 20:51:26.331171 2459 tablet_bootstrap.cc:492] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Bootstrap starting.
I20250112 20:51:26.332758 2170 heartbeater.cc:502] Master 127.0.104.62:45501 was elected leader, sending a full tablet report...
I20250112 20:51:26.333109 2459 tablet_bootstrap.cc:654] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.337339 2461 ts_tablet_manager.cc:1428] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa: Time spent starting tablet: real 0.021s user 0.013s sys 0.000s
I20250112 20:51:26.337824 2442 heartbeater.cc:502] Master 127.0.104.62:45501 was elected leader, sending a full tablet report...
I20250112 20:51:26.343541 2461 tablet_bootstrap.cc:492] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa: Bootstrap starting.
I20250112 20:51:26.346093 2461 tablet_bootstrap.cc:654] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.347851 2460 ts_tablet_manager.cc:1428] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9: Time spent starting tablet: real 0.033s user 0.020s sys 0.013s
I20250112 20:51:26.348768 2460 tablet_bootstrap.cc:492] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9: Bootstrap starting.
I20250112 20:51:26.349331 2461 tablet_bootstrap.cc:492] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa: No bootstrap required, opened a new log
I20250112 20:51:26.349547 2461 ts_tablet_manager.cc:1397] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250112 20:51:26.350549 2460 tablet_bootstrap.cc:654] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.350445 2461 raft_consensus.cc:357] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.350759 2461 raft_consensus.cc:383] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.350906 2461 raft_consensus.cc:738] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.351266 2461 consensus_queue.cc:260] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.352761 2461 ts_tablet_manager.cc:1428] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:26.354912 2460 tablet_bootstrap.cc:492] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9: No bootstrap required, opened a new log
I20250112 20:51:26.355129 2460 ts_tablet_manager.cc:1397] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20250112 20:51:26.355360 2459 tablet_bootstrap.cc:492] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: No bootstrap required, opened a new log
I20250112 20:51:26.355566 2459 ts_tablet_manager.cc:1397] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.012s
I20250112 20:51:26.356222 2460 raft_consensus.cc:357] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.356561 2460 raft_consensus.cc:383] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.356478 2459 raft_consensus.cc:357] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.356683 2460 raft_consensus.cc:738] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.356756 2459 raft_consensus.cc:383] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.356858 2459 raft_consensus.cc:738] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7299d0bd6a4d9f8f7212ada1f7da3a, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.357113 2459 consensus_queue.cc:260] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.357124 2460 consensus_queue.cc:260] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.357717 2460 ts_tablet_manager.cc:1428] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:26.357729 2459 ts_tablet_manager.cc:1428] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:26.371943 2467 raft_consensus.cc:491] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.372180 2467 raft_consensus.cc:513] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.373852 2467 leader_election.cc:290] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751), 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097)
I20250112 20:51:26.380147 2125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231f016eec6148bd8145f2ce5eab971c" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9" is_pre_election: true
I20250112 20:51:26.380705 2125 raft_consensus.cc:2463] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:26.380848 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231f016eec6148bd8145f2ce5eab971c" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" is_pre_election: true
I20250112 20:51:26.381312 2261 raft_consensus.cc:2463] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:26.381307 2323 leader_election.cc:304] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:26.381630 2467 raft_consensus.cc:2798] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.381748 2467 raft_consensus.cc:491] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.381846 2467 raft_consensus.cc:3054] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.384088 2467 raft_consensus.cc:513] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.384783 2467 leader_election.cc:290] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Requested vote from peers 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751), 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097)
I20250112 20:51:26.385165 2125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231f016eec6148bd8145f2ce5eab971c" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9"
I20250112 20:51:26.385274 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "231f016eec6148bd8145f2ce5eab971c" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
I20250112 20:51:26.385390 2125 raft_consensus.cc:3054] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.385481 2261 raft_consensus.cc:3054] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.388615 2261 raft_consensus.cc:2463] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:26.388610 2125 raft_consensus.cc:2463] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:26.388974 2323 leader_election.cc:304] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:26.389250 2467 raft_consensus.cc:2798] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.389703 2467 raft_consensus.cc:695] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [term 1 LEADER]: Becoming Leader. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Running, Role: LEADER
I20250112 20:51:26.390216 2467 consensus_queue.cc:237] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.394660 1966 catalog_manager.cc:5526] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3). New cstate: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:26.463496 2443 tablet.cc:2367] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:26.510799 2467 raft_consensus.cc:491] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.511029 2467 raft_consensus.cc:513] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.511648 2467 leader_election.cc:290] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097), 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751)
I20250112 20:51:26.512162 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b58592de9fb45749b79a21c10a7ca65" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" is_pre_election: true
I20250112 20:51:26.512183 2125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b58592de9fb45749b79a21c10a7ca65" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9" is_pre_election: true
I20250112 20:51:26.512396 2125 raft_consensus.cc:2463] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:26.512414 2261 raft_consensus.cc:2463] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:26.512848 2323 leader_election.cc:304] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:26.513141 2467 raft_consensus.cc:2798] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.513243 2467 raft_consensus.cc:491] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.513350 2467 raft_consensus.cc:3054] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.516251 2467 raft_consensus.cc:513] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.516897 2467 leader_election.cc:290] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Requested vote from peers 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097), 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751)
I20250112 20:51:26.517390 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b58592de9fb45749b79a21c10a7ca65" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
I20250112 20:51:26.517416 2125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b58592de9fb45749b79a21c10a7ca65" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9"
I20250112 20:51:26.517633 2125 raft_consensus.cc:3054] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.517657 2261 raft_consensus.cc:3054] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:26.519505 2307 tablet.cc:2367] T 231f016eec6148bd8145f2ce5eab971c P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:26.520480 2125 raft_consensus.cc:2463] T 2b58592de9fb45749b79a21c10a7ca65 P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:26.520480 2261 raft_consensus.cc:2463] T 2b58592de9fb45749b79a21c10a7ca65 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:26.520951 2324 leader_election.cc:304] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7299d0bd6a4d9f8f7212ada1f7da3a, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:26.521224 2467 raft_consensus.cc:2798] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.521529 2467 raft_consensus.cc:695] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [term 1 LEADER]: Becoming Leader. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Running, Role: LEADER
I20250112 20:51:26.521852 2467 consensus_queue.cc:237] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } }
I20250112 20:51:26.524261 1966 catalog_manager.cc:5526] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3). New cstate: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:26.575879 2171 tablet.cc:2367] T 231f016eec6148bd8145f2ce5eab971c P 23aa8791e86440869c09782df3a8d6c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:26.617417 1966 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47570:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001\003\000\000\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:26.618324 1966 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:26.624424 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 847698bc16ae49cdaf515cdad05f59a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 3
I20250112 20:51:26.624392 2240 tablet_service.cc:1467] Processing CreateTablet for tablet 847698bc16ae49cdaf515cdad05f59a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 3
I20250112 20:51:26.625422 2103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 847698bc16ae49cdaf515cdad05f59a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.625422 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 847698bc16ae49cdaf515cdad05f59a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.629137 2371 tablet_service.cc:1467] Processing CreateTablet for tablet 847698bc16ae49cdaf515cdad05f59a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 3
I20250112 20:51:26.630077 2371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 847698bc16ae49cdaf515cdad05f59a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:26.632683 2459 tablet_bootstrap.cc:492] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Bootstrap starting.
I20250112 20:51:26.632731 2460 tablet_bootstrap.cc:492] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: Bootstrap starting.
I20250112 20:51:26.634320 2459 tablet_bootstrap.cc:654] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.634616 2460 tablet_bootstrap.cc:654] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.636857 2460 tablet_bootstrap.cc:492] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: No bootstrap required, opened a new log
I20250112 20:51:26.637032 2460 ts_tablet_manager.cc:1397] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:51:26.637081 2461 tablet_bootstrap.cc:492] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: Bootstrap starting.
I20250112 20:51:26.638003 2460 raft_consensus.cc:357] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.638224 2460 raft_consensus.cc:383] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.638311 2460 raft_consensus.cc:738] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.638579 2461 tablet_bootstrap.cc:654] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:26.638654 2460 consensus_queue.cc:260] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.639261 2460 ts_tablet_manager.cc:1428] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:26.639171 2466 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.639564 2466 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:26.640228 2459 tablet_bootstrap.cc:492] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: No bootstrap required, opened a new log
I20250112 20:51:26.640416 2459 ts_tablet_manager.cc:1397] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20250112 20:51:26.641412 2459 raft_consensus.cc:357] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.641578 2461 tablet_bootstrap.cc:492] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: No bootstrap required, opened a new log
I20250112 20:51:26.641587 2459 raft_consensus.cc:383] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.641655 2459 raft_consensus.cc:738] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7299d0bd6a4d9f8f7212ada1f7da3a, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.641739 2461 ts_tablet_manager.cc:1397] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:51:26.641927 2459 consensus_queue.cc:260] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.642372 2465 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.642624 2465 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:26.642558 2461 raft_consensus.cc:357] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.642781 2461 raft_consensus.cc:383] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:26.642858 2461 raft_consensus.cc:738] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Initialized, Role: FOLLOWER
I20250112 20:51:26.643085 2461 consensus_queue.cc:260] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.644207 2459 ts_tablet_manager.cc:1428] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:26.644387 2467 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.644616 2467 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:26.645460 2461 ts_tablet_manager.cc:1428] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:26.806670 2467 raft_consensus.cc:491] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:26.806891 2467 raft_consensus.cc:513] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.807641 2467 leader_election.cc:290] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751), 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097)
I20250112 20:51:26.808207 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847698bc16ae49cdaf515cdad05f59a9" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" is_pre_election: true
I20250112 20:51:26.808220 2125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847698bc16ae49cdaf515cdad05f59a9" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9" is_pre_election: true
I20250112 20:51:26.808454 2125 raft_consensus.cc:2463] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:26.808477 2261 raft_consensus.cc:2463] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:26.808888 2324 leader_election.cc:304] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7299d0bd6a4d9f8f7212ada1f7da3a, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:26.809165 2467 raft_consensus.cc:2798] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:26.809273 2467 raft_consensus.cc:491] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:26.809360 2467 raft_consensus.cc:3054] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.811100 2467 raft_consensus.cc:513] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.811694 2467 leader_election.cc:290] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Requested vote from peers 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751), 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097)
I20250112 20:51:26.812090 2125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847698bc16ae49cdaf515cdad05f59a9" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9"
I20250112 20:51:26.812181 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847698bc16ae49cdaf515cdad05f59a9" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
I20250112 20:51:26.812292 2125 raft_consensus.cc:3054] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.812350 2261 raft_consensus.cc:3054] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:26.815074 2261 raft_consensus.cc:2463] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:26.815074 2125 raft_consensus.cc:2463] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:26.815531 2324 leader_election.cc:304] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, 2d7299d0bd6a4d9f8f7212ada1f7da3a, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:26.815826 2467 raft_consensus.cc:2798] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:26.815943 2467 raft_consensus.cc:695] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [term 1 LEADER]: Becoming Leader. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Running, Role: LEADER
I20250112 20:51:26.816221 2467 consensus_queue.cc:237] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:26.817634 2472 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: New leader d45e7b3cee7541df91a1450d5d8054fa. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.817822 2472 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:26.818367 2486 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:26.818630 2486 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:26.819100 1966 catalog_manager.cc:5526] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3). New cstate: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:26.880046 2261 raft_consensus.cc:1270] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Refusing update from remote peer d45e7b3cee7541df91a1450d5d8054fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:26.880039 2125 raft_consensus.cc:1270] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Refusing update from remote peer d45e7b3cee7541df91a1450d5d8054fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:26.881048 2467 consensus_queue.cc:1035] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:26.881485 2472 consensus_queue.cc:1035] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:26.881525 2490 consensus_queue.cc:1035] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:26.906679 2465 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: New leader d45e7b3cee7541df91a1450d5d8054fa. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.906955 2465 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:26.908814 2466 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: New leader d45e7b3cee7541df91a1450d5d8054fa. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.909075 2466 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:26.909752 2472 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.910001 2472 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:26.909950 2467 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.910187 2467 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:26.911003 2490 consensus_queue.cc:1035] T 231f016eec6148bd8145f2ce5eab971c P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:26.912441 2494 mvcc.cc:204] Tried to move back new op lower bound from 7113584995851276288 to 7113584995601567744. Current Snapshot: MvccSnapshot[applied={T|T < 7113584995851276288}]
I20250112 20:51:26.938014 2490 consensus_queue.cc:1035] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:26.940928 2466 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.941181 2466 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:26.945752 2465 tablet_replica.cc:440] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:26.945994 2465 tablet_replica.cc:443] T 847698bc16ae49cdaf515cdad05f59a9 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:26.955602 2467 consensus_queue.cc:1035] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.150837 1965 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47570:
table { table_name: "kudu_system.kudu_transactions" } alter_schema_steps { type: ADD_RANGE_PARTITION add_range_partition { range_bounds { rows: "<redacted>""\006\001\003\000\000\000\000\000\000\000\007\001\006\000\000\000\000\000\000\000" indirect_data: "<redacted>""" } } } schema { columns { name: "txn_id" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "entry_type" type: INT8 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "identifier" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "metadata" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: false
W20250112 20:51:27.155897 1965 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 685bbe4f1a734b95934611e2aee1d301 has no consensus state
I20250112 20:51:27.160415 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 685bbe4f1a734b95934611e2aee1d301 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 3 <= VALUES < 6
I20250112 20:51:27.160583 2240 tablet_service.cc:1467] Processing CreateTablet for tablet 685bbe4f1a734b95934611e2aee1d301 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 3 <= VALUES < 6
I20250112 20:51:27.160975 2103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685bbe4f1a734b95934611e2aee1d301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.160995 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685bbe4f1a734b95934611e2aee1d301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.161994 2372 tablet_service.cc:1467] Processing CreateTablet for tablet 685bbe4f1a734b95934611e2aee1d301 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 3 <= VALUES < 6
I20250112 20:51:27.162566 2372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 685bbe4f1a734b95934611e2aee1d301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.175941 2512 tablet_bootstrap.cc:492] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: Bootstrap starting.
I20250112 20:51:27.176209 2513 tablet_bootstrap.cc:492] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Bootstrap starting.
I20250112 20:51:27.178290 2512 tablet_bootstrap.cc:654] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.178431 2513 tablet_bootstrap.cc:654] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.182444 2514 tablet_bootstrap.cc:492] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: Bootstrap starting.
I20250112 20:51:27.184669 2514 tablet_bootstrap.cc:654] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.186524 2512 tablet_bootstrap.cc:492] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: No bootstrap required, opened a new log
I20250112 20:51:27.186523 2513 tablet_bootstrap.cc:492] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: No bootstrap required, opened a new log
I20250112 20:51:27.186693 2513 ts_tablet_manager.cc:1397] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250112 20:51:27.186686 2512 ts_tablet_manager.cc:1397] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250112 20:51:27.187505 2512 raft_consensus.cc:357] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.187505 2513 raft_consensus.cc:357] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.187741 2513 raft_consensus.cc:383] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.187857 2512 raft_consensus.cc:383] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.187855 2513 raft_consensus.cc:738] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7299d0bd6a4d9f8f7212ada1f7da3a, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.188006 2512 raft_consensus.cc:738] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.188129 2513 consensus_queue.cc:260] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.188354 2512 consensus_queue.cc:260] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.188644 2465 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.188867 2465 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.188827 2503 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.189103 2503 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.189266 2512 ts_tablet_manager.cc:1428] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250112 20:51:27.189903 2513 ts_tablet_manager.cc:1428] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:51:27.193334 1966 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9] alter complete (version 0)
I20250112 20:51:27.197962 2514 tablet_bootstrap.cc:492] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: No bootstrap required, opened a new log
I20250112 20:51:27.198148 2514 ts_tablet_manager.cc:1397] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20250112 20:51:27.199323 2514 raft_consensus.cc:357] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.199597 2514 raft_consensus.cc:383] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.199697 2514 raft_consensus.cc:738] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.199940 2514 consensus_queue.cc:260] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.200345 2472 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.200630 2472 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.201138 2514 ts_tablet_manager.cc:1428] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20250112 20:51:27.208716 1952 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 685bbe4f1a734b95934611e2aee1d301 (table kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9], current schema version=0) failed: Not found: failed to reset TS proxy: No leader found: Tablet 685bbe4f1a734b95934611e2aee1d301 consensus state has no leader
I20250112 20:51:27.289271 2472 raft_consensus.cc:491] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.289553 2472 raft_consensus.cc:513] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.290503 2472 leader_election.cc:290] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751), 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097)
I20250112 20:51:27.290958 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685bbe4f1a734b95934611e2aee1d301" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9" is_pre_election: true
I20250112 20:51:27.291213 2123 raft_consensus.cc:2463] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:27.291214 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685bbe4f1a734b95934611e2aee1d301" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" is_pre_election: true
I20250112 20:51:27.291472 2261 raft_consensus.cc:2463] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 0.
I20250112 20:51:27.291666 2323 leader_election.cc:304] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:27.291975 2472 raft_consensus.cc:2798] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:27.292070 2472 raft_consensus.cc:491] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:27.292179 2472 raft_consensus.cc:3054] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.294247 2472 raft_consensus.cc:513] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.295149 2472 leader_election.cc:290] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Requested vote from peers 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751), 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097)
I20250112 20:51:27.295347 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685bbe4f1a734b95934611e2aee1d301" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
I20250112 20:51:27.295305 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "685bbe4f1a734b95934611e2aee1d301" candidate_uuid: "d45e7b3cee7541df91a1450d5d8054fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23aa8791e86440869c09782df3a8d6c9"
I20250112 20:51:27.295502 2261 raft_consensus.cc:3054] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.295512 2123 raft_consensus.cc:3054] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.297433 2261 raft_consensus.cc:2463] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:27.297842 2324 leader_election.cc:304] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d7299d0bd6a4d9f8f7212ada1f7da3a, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:27.298117 2472 raft_consensus.cc:2798] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:27.298300 2472 raft_consensus.cc:695] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [term 1 LEADER]: Becoming Leader. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Running, Role: LEADER
I20250112 20:51:27.298662 2123 raft_consensus.cc:2463] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d45e7b3cee7541df91a1450d5d8054fa in term 1.
I20250112 20:51:27.298568 2472 consensus_queue.cc:237] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.300180 2490 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: New leader d45e7b3cee7541df91a1450d5d8054fa. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.300603 2490 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:27.300822 2486 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:27.300992 2486 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:27.303309 1966 catalog_manager.cc:5526] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3). New cstate: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:27.321352 2261 raft_consensus.cc:1270] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Refusing update from remote peer d45e7b3cee7541df91a1450d5d8054fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:27.321419 2123 raft_consensus.cc:1270] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Refusing update from remote peer d45e7b3cee7541df91a1450d5d8054fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:27.322027 2472 consensus_queue.cc:1035] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.322327 2490 consensus_queue.cc:1035] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.325146 2465 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: New leader d45e7b3cee7541df91a1450d5d8054fa. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.325400 2465 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.326902 2490 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.327108 2490 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:27.327625 2489 mvcc.cc:204] Tried to move back new op lower bound from 7113584997662654464 to 7113584997578022912. Current Snapshot: MvccSnapshot[applied={T|T < 7113584997662654464 or (T in {7113584997662654464})}]
I20250112 20:51:27.328821 2465 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.329047 2465 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.330951 2503 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: New leader d45e7b3cee7541df91a1450d5d8054fa. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.331238 2503 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.334045 2472 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.334273 2472 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:27.336045 2503 tablet_replica.cc:440] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d45e7b3cee7541df91a1450d5d8054fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.336264 2503 tablet_replica.cc:443] T 685bbe4f1a734b95934611e2aee1d301 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.473392 1966 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47570:
table { table_name: "kudu_system.kudu_transactions" } alter_schema_steps { type: ADD_RANGE_PARTITION add_range_partition { range_bounds { rows: "<redacted>""\006\001\006\000\000\000\000\000\000\000\007\001\t\000\000\000\000\000\000\000" indirect_data: "<redacted>""" } } } schema { columns { name: "txn_id" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "entry_type" type: INT8 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "identifier" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "metadata" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: false
W20250112 20:51:27.477195 1966 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2b5aaeb0c0fb455b910097e3e83977bb has no consensus state
I20250112 20:51:27.481537 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5aaeb0c0fb455b910097e3e83977bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 6 <= VALUES < 9
I20250112 20:51:27.481679 2371 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5aaeb0c0fb455b910097e3e83977bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 6 <= VALUES < 9
I20250112 20:51:27.481655 2240 tablet_service.cc:1467] Processing CreateTablet for tablet 2b5aaeb0c0fb455b910097e3e83977bb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 6 <= VALUES < 9
I20250112 20:51:27.482052 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5aaeb0c0fb455b910097e3e83977bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.482064 2103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5aaeb0c0fb455b910097e3e83977bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.482069 2371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b5aaeb0c0fb455b910097e3e83977bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.486335 2513 tablet_bootstrap.cc:492] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Bootstrap starting.
I20250112 20:51:27.487620 2513 tablet_bootstrap.cc:654] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.488282 2514 tablet_bootstrap.cc:492] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: Bootstrap starting.
I20250112 20:51:27.489727 2513 tablet_bootstrap.cc:492] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: No bootstrap required, opened a new log
I20250112 20:51:27.489820 2512 tablet_bootstrap.cc:492] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: Bootstrap starting.
I20250112 20:51:27.489881 2513 ts_tablet_manager.cc:1397] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250112 20:51:27.490028 2514 tablet_bootstrap.cc:654] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.490600 2513 raft_consensus.cc:357] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.490846 2513 raft_consensus.cc:383] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.490916 2513 raft_consensus.cc:738] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7299d0bd6a4d9f8f7212ada1f7da3a, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.491158 2513 consensus_queue.cc:260] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.491500 2512 tablet_bootstrap.cc:654] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.491537 2465 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.491716 2465 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.492997 2513 ts_tablet_manager.cc:1428] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:27.493872 2512 tablet_bootstrap.cc:492] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: No bootstrap required, opened a new log
I20250112 20:51:27.494103 2512 ts_tablet_manager.cc:1397] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20250112 20:51:27.494892 2512 raft_consensus.cc:357] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.495465 2512 raft_consensus.cc:383] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.495594 2512 raft_consensus.cc:738] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.495867 2512 consensus_queue.cc:260] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.495966 1966 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9] alter complete (version 0)
I20250112 20:51:27.496318 2514 tablet_bootstrap.cc:492] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: No bootstrap required, opened a new log
I20250112 20:51:27.496268 2503 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.496484 2503 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.496482 2512 ts_tablet_manager.cc:1428] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:51:27.496516 2514 ts_tablet_manager.cc:1397] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250112 20:51:27.497262 2514 raft_consensus.cc:357] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.497443 2514 raft_consensus.cc:383] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.497517 2514 raft_consensus.cc:738] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.497761 2514 consensus_queue.cc:260] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.498108 2467 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.498243 2514 ts_tablet_manager.cc:1428] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:27.498289 2467 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.578974 2503 raft_consensus.cc:491] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:27.579159 2503 raft_consensus.cc:513] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.580637 2503 leader_election.cc:290] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097), d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3:41043)
I20250112 20:51:27.598027 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5aaeb0c0fb455b910097e3e83977bb" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" is_pre_election: true
I20250112 20:51:27.598090 2397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5aaeb0c0fb455b910097e3e83977bb" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45e7b3cee7541df91a1450d5d8054fa" is_pre_election: true
I20250112 20:51:27.598263 2261 raft_consensus.cc:2463] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 0.
I20250112 20:51:27.598453 2397 raft_consensus.cc:2463] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 0.
I20250112 20:51:27.598886 2052 leader_election.cc:304] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, 2d7299d0bd6a4d9f8f7212ada1f7da3a; no voters:
I20250112 20:51:27.599341 2503 raft_consensus.cc:2798] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:27.599486 2503 raft_consensus.cc:491] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:27.599592 2503 raft_consensus.cc:3054] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.601600 2503 raft_consensus.cc:513] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.602185 2503 leader_election.cc:290] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097), d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3:41043)
I20250112 20:51:27.602648 2261 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5aaeb0c0fb455b910097e3e83977bb" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
I20250112 20:51:27.602679 2397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b5aaeb0c0fb455b910097e3e83977bb" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45e7b3cee7541df91a1450d5d8054fa"
I20250112 20:51:27.602806 2261 raft_consensus.cc:3054] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.602876 2397 raft_consensus.cc:3054] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:27.604715 2261 raft_consensus.cc:2463] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 1.
I20250112 20:51:27.604997 2397 raft_consensus.cc:2463] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 1.
I20250112 20:51:27.605082 2052 leader_election.cc:304] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, 2d7299d0bd6a4d9f8f7212ada1f7da3a; no voters:
I20250112 20:51:27.605358 2503 raft_consensus.cc:2798] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:27.605738 2503 raft_consensus.cc:695] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Running, Role: LEADER
I20250112 20:51:27.606191 2503 consensus_queue.cc:237] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.607877 2528 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: New leader 23aa8791e86440869c09782df3a8d6c9. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.608239 2528 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:27.609015 2530 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:27.609242 2530 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:27.609776 1967 catalog_manager.cc:5526] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:27.626273 2261 raft_consensus.cc:1270] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Refusing update from remote peer 23aa8791e86440869c09782df3a8d6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:27.626511 2397 raft_consensus.cc:1270] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Refusing update from remote peer 23aa8791e86440869c09782df3a8d6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:27.627009 2503 consensus_queue.cc:1035] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.627275 2528 consensus_queue.cc:1035] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:27.630232 2467 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: New leader 23aa8791e86440869c09782df3a8d6c9. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.630467 2467 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.632011 2503 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.632249 2503 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:27.635588 2467 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.635845 2467 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.638149 2465 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: New leader 23aa8791e86440869c09782df3a8d6c9. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.638413 2465 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.641322 2528 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.641649 2528 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:27.643088 2465 tablet_replica.cc:440] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.643319 2465 tablet_replica.cc:443] T 2b5aaeb0c0fb455b910097e3e83977bb P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.837726 1965 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47570:
table { table_name: "kudu_system.kudu_transactions" } alter_schema_steps { type: ADD_RANGE_PARTITION add_range_partition { range_bounds { rows: "<redacted>""\006\001\t\000\000\000\000\000\000\000\007\001\014\000\000\000\000\000\000\000" indirect_data: "<redacted>""" } } } schema { columns { name: "txn_id" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "entry_type" type: INT8 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "identifier" type: STRING is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { name: "metadata" type: STRING is_key: false is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } modify_external_catalogs: false
W20250112 20:51:27.842476 1965 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75e1f64c5e4e4295848cf1adc32ab447 has no consensus state
I20250112 20:51:27.846846 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 75e1f64c5e4e4295848cf1adc32ab447 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 9 <= VALUES < 12
I20250112 20:51:27.847056 2240 tablet_service.cc:1467] Processing CreateTablet for tablet 75e1f64c5e4e4295848cf1adc32ab447 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 9 <= VALUES < 12
I20250112 20:51:27.847257 2372 tablet_service.cc:1467] Processing CreateTablet for tablet 75e1f64c5e4e4295848cf1adc32ab447 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9]), partition=RANGE (txn_id) PARTITION 9 <= VALUES < 12
I20250112 20:51:27.847505 2240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e1f64c5e4e4295848cf1adc32ab447. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.847677 2372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e1f64c5e4e4295848cf1adc32ab447. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.847823 2103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75e1f64c5e4e4295848cf1adc32ab447. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:27.853852 2513 tablet_bootstrap.cc:492] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Bootstrap starting.
I20250112 20:51:27.854049 2514 tablet_bootstrap.cc:492] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: Bootstrap starting.
I20250112 20:51:27.854049 2512 tablet_bootstrap.cc:492] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: Bootstrap starting.
I20250112 20:51:27.855702 2513 tablet_bootstrap.cc:654] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.855859 2514 tablet_bootstrap.cc:654] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.855859 2512 tablet_bootstrap.cc:654] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:27.868248 2514 tablet_bootstrap.cc:492] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: No bootstrap required, opened a new log
I20250112 20:51:27.868602 2514 ts_tablet_manager.cc:1397] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20250112 20:51:27.869776 2514 raft_consensus.cc:357] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.870177 2514 raft_consensus.cc:383] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.870337 2514 raft_consensus.cc:738] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d45e7b3cee7541df91a1450d5d8054fa, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.870739 2513 tablet_bootstrap.cc:492] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: No bootstrap required, opened a new log
I20250112 20:51:27.870900 2513 ts_tablet_manager.cc:1397] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.008s
I20250112 20:51:27.870740 2514 consensus_queue.cc:260] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.871380 2467 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.871615 2467 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.872824 2513 raft_consensus.cc:357] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.873039 2513 raft_consensus.cc:383] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.873131 2513 raft_consensus.cc:738] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7299d0bd6a4d9f8f7212ada1f7da3a, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.873361 2514 ts_tablet_manager.cc:1428] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:27.873351 2513 consensus_queue.cc:260] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.873914 2513 ts_tablet_manager.cc:1428] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:27.874594 2465 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.874773 2465 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.876794 1965 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 7a11a4cd17aa445983ef5ae082e2e413: kudu_system.kudu_transactions [id=1d08666e01c14f79b0fc0c00212fa0c9] alter complete (version 0)
I20250112 20:51:27.877151 2512 tablet_bootstrap.cc:492] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: No bootstrap required, opened a new log
I20250112 20:51:27.877305 2512 ts_tablet_manager.cc:1397] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.008s
I20250112 20:51:27.878101 2512 raft_consensus.cc:357] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.878350 2512 raft_consensus.cc:383] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:27.878468 2512 raft_consensus.cc:738] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Initialized, Role: FOLLOWER
I20250112 20:51:27.880179 2512 consensus_queue.cc:260] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:27.880599 2544 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:27.880791 2544 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:27.881280 2512 ts_tablet_manager.cc:1428] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250112 20:51:28.048417 2544 raft_consensus.cc:491] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:28.048651 2544 raft_consensus.cc:513] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:28.049364 2544 leader_election.cc:290] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097), d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3:41043)
I20250112 20:51:28.049937 2260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e1f64c5e4e4295848cf1adc32ab447" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" is_pre_election: true
I20250112 20:51:28.049966 2397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e1f64c5e4e4295848cf1adc32ab447" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45e7b3cee7541df91a1450d5d8054fa" is_pre_election: true
I20250112 20:51:28.050189 2260 raft_consensus.cc:2463] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 0.
I20250112 20:51:28.050189 2397 raft_consensus.cc:2463] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 0.
I20250112 20:51:28.050616 2053 leader_election.cc:304] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:28.050895 2544 raft_consensus.cc:2798] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:28.051017 2544 raft_consensus.cc:491] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:28.051105 2544 raft_consensus.cc:3054] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:28.053956 2544 raft_consensus.cc:513] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:28.054734 2544 leader_election.cc:290] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097), d45e7b3cee7541df91a1450d5d8054fa (127.0.104.3:41043)
I20250112 20:51:28.055279 2397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e1f64c5e4e4295848cf1adc32ab447" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d45e7b3cee7541df91a1450d5d8054fa"
I20250112 20:51:28.055279 2260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75e1f64c5e4e4295848cf1adc32ab447" candidate_uuid: "23aa8791e86440869c09782df3a8d6c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a"
I20250112 20:51:28.055527 2260 raft_consensus.cc:3054] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:28.055523 2397 raft_consensus.cc:3054] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:28.058560 2260 raft_consensus.cc:2463] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 1.
I20250112 20:51:28.058560 2397 raft_consensus.cc:2463] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23aa8791e86440869c09782df3a8d6c9 in term 1.
I20250112 20:51:28.059046 2053 leader_election.cc:304] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23aa8791e86440869c09782df3a8d6c9, d45e7b3cee7541df91a1450d5d8054fa; no voters:
I20250112 20:51:28.059378 2544 raft_consensus.cc:2798] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:28.059571 2544 raft_consensus.cc:695] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [term 1 LEADER]: Becoming Leader. State: Replica: 23aa8791e86440869c09782df3a8d6c9, State: Running, Role: LEADER
I20250112 20:51:28.059886 2544 consensus_queue.cc:237] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } }
I20250112 20:51:28.061697 2503 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: New leader 23aa8791e86440869c09782df3a8d6c9. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:28.061915 2503 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:28.062222 2530 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:28.062464 2530 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:28.063598 1967 catalog_manager.cc:5526] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:28.081326 2260 raft_consensus.cc:1270] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a [term 1 FOLLOWER]: Refusing update from remote peer 23aa8791e86440869c09782df3a8d6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:28.081326 2397 raft_consensus.cc:1270] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa [term 1 FOLLOWER]: Refusing update from remote peer 23aa8791e86440869c09782df3a8d6c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:28.081882 2544 consensus_queue.cc:1035] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:28.082165 2503 consensus_queue.cc:1035] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:28.084671 2467 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: New leader 23aa8791e86440869c09782df3a8d6c9. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:28.084939 2467 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:28.086426 2544 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:28.086694 2544 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:28.088784 2467 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:28.089020 2467 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P d45e7b3cee7541df91a1450d5d8054fa: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:28.093089 2465 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: New leader 23aa8791e86440869c09782df3a8d6c9. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:28.093304 2465 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:28.094372 2528 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:28.094625 2528 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P 23aa8791e86440869c09782df3a8d6c9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:28.099754 2465 tablet_replica.cc:440] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23aa8791e86440869c09782df3a8d6c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23aa8791e86440869c09782df3a8d6c9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38751 } } peers { permanent_uuid: "2d7299d0bd6a4d9f8f7212ada1f7da3a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41097 } } peers { permanent_uuid: "d45e7b3cee7541df91a1450d5d8054fa" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41043 } } }
I20250112 20:51:28.100032 2465 tablet_replica.cc:443] T 75e1f64c5e4e4295848cf1adc32ab447 P 2d7299d0bd6a4d9f8f7212ada1f7da3a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:28.159971 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 2036
I20250112 20:51:28.161033 2165 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:28.323235 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 2036
I20250112 20:51:28.345535 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 2172
W20250112 20:51:28.345657 2189 connection.cc:537] server connection from 127.0.104.1:44799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 20:51:28.346596 2301 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:28.445004 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 2172
W20250112 20:51:28.452668 2323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:38751: connect: Connection refused (error 111)
W20250112 20:51:28.454352 2323 consensus_peers.cc:487] T 2b58592de9fb45749b79a21c10a7ca65 P d45e7b3cee7541df91a1450d5d8054fa -> Peer 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751): Couldn't send request to peer 23aa8791e86440869c09782df3a8d6c9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:28.455365 2323 consensus_peers.cc:487] T 685bbe4f1a734b95934611e2aee1d301 P d45e7b3cee7541df91a1450d5d8054fa -> Peer 23aa8791e86440869c09782df3a8d6c9 (127.0.104.1:38751): Couldn't send request to peer 23aa8791e86440869c09782df3a8d6c9. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:38751: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.471302 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 2308
W20250112 20:51:28.471987 2324 consensus_peers.cc:487] T 847698bc16ae49cdaf515cdad05f59a9 P d45e7b3cee7541df91a1450d5d8054fa -> Peer 2d7299d0bd6a4d9f8f7212ada1f7da3a (127.0.104.2:41097): Couldn't send request to peer 2d7299d0bd6a4d9f8f7212ada1f7da3a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.2:41097: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:28.472929 2437 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:28.626048 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 2308
I20250112 20:51:28.650862 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 1936
I20250112 20:51:28.651989 2007 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:28.767473 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 1936
2025-01-12T20:51:28Z chronyd exiting
[ OK ] TxnWriteOpsITest.TestWriteToNewRangeOfTxnIds (4445 ms)
[ RUN ] TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit
2025-01-12T20:51:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:28Z Disabled control of system clock
I20250112 20:51:28.836165 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:45417
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:45417
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20250112 20:51:29.158704 2561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:29.159054 2561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:29.159140 2561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:29.167369 2561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:29.167490 2561 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:29.167528 2561 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20250112 20:51:29.167606 2561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:29.167677 2561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:29.167763 2561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:29.177182 2561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:45417
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:45417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:29.178763 2561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:29.180451 2561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:29.190513 2566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:29.190547 2569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:29.190572 2567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:29.191488 2561 server_base.cc:1034] running on GCE node
I20250112 20:51:29.192260 2561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:29.193778 2561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:29.195006 2561 hybrid_clock.cc:648] HybridClock initialized: now 1736715089194946 us; error 40 us; skew 500 ppm
I20250112 20:51:29.195327 2561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:29.197688 2561 webserver.cc:458] Webserver started at http://127.0.104.62:34087/ using document root <none> and password file <none>
I20250112 20:51:29.198413 2561 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:29.198534 2561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:29.198834 2561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:29.201543 2561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "0ccbb217dee2459e8333de51ba8cd5e4"
format_stamp: "Formatted at 2025-01-12 20:51:29 on dist-test-slave-3gql"
I20250112 20:51:29.202327 2561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "0ccbb217dee2459e8333de51ba8cd5e4"
format_stamp: "Formatted at 2025-01-12 20:51:29 on dist-test-slave-3gql"
I20250112 20:51:29.207631 2561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250112 20:51:29.210952 2575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:29.212419 2561 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250112 20:51:29.212613 2561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "0ccbb217dee2459e8333de51ba8cd5e4"
format_stamp: "Formatted at 2025-01-12 20:51:29 on dist-test-slave-3gql"
I20250112 20:51:29.212810 2561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:29.251869 2561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:29.252821 2561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:29.253115 2561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:29.274215 2561 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:45417
I20250112 20:51:29.274250 2636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:45417 every 8 connection(s)
I20250112 20:51:29.275642 2561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:29.279531 2637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:29.284730 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 2561
I20250112 20:51:29.284945 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:29.290431 2637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4: Bootstrap starting.
I20250112 20:51:29.293658 2637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:29.294886 2637 log.cc:826] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:29.297987 2637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4: No bootstrap required, opened a new log
I20250112 20:51:29.303830 2637 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45417 } }
I20250112 20:51:29.304177 2637 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:29.304294 2637 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ccbb217dee2459e8333de51ba8cd5e4, State: Initialized, Role: FOLLOWER
I20250112 20:51:29.304925 2637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45417 } }
I20250112 20:51:29.305148 2637 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:29.305299 2637 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:29.305461 2637 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:29.307425 2637 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45417 } }
I20250112 20:51:29.307893 2637 leader_election.cc:304] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ccbb217dee2459e8333de51ba8cd5e4; no voters:
I20250112 20:51:29.308422 2637 leader_election.cc:290] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:29.308593 2642 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:29.309188 2642 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [term 1 LEADER]: Becoming Leader. State: Replica: 0ccbb217dee2459e8333de51ba8cd5e4, State: Running, Role: LEADER
I20250112 20:51:29.309736 2642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45417 } }
I20250112 20:51:29.310547 2637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:29.312662 2644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ccbb217dee2459e8333de51ba8cd5e4. Latest consensus state: current_term: 1 leader_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45417 } } }
I20250112 20:51:29.313078 2644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:29.313050 2643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ccbb217dee2459e8333de51ba8cd5e4" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45417 } } }
I20250112 20:51:29.313331 2643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:29.315156 2650 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:29.319744 2650 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:29.336005 2650 catalog_manager.cc:1348] Generated new cluster ID: 013241eb61224cf796ea6d7f48c12ce8
I20250112 20:51:29.336207 2650 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:29.351505 2650 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:29.353157 2650 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:29.365183 2650 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 0ccbb217dee2459e8333de51ba8cd5e4: Generated new TSK 0
I20250112 20:51:29.366266 2650 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:29.376509 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45417
--builtin_ntp_servers=127.0.104.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:29.380645 2592 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45162:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:29.392030 2637 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20250112 20:51:29.728061 2666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:29.728359 2666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:29.728492 2666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:29.736392 2666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:29.736583 2666 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:29.736750 2666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:29.749541 2666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:45417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:29.751155 2666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:29.752916 2666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:29.764520 2671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:29.764595 2672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:29.764572 2674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:29.766222 2666 server_base.cc:1034] running on GCE node
I20250112 20:51:29.766952 2666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:29.767915 2666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:29.769114 2666 hybrid_clock.cc:648] HybridClock initialized: now 1736715089769062 us; error 45 us; skew 500 ppm
I20250112 20:51:29.769445 2666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:29.771636 2666 webserver.cc:458] Webserver started at http://127.0.104.1:44701/ using document root <none> and password file <none>
I20250112 20:51:29.772359 2666 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:29.772471 2666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:29.772799 2666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:29.775440 2666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "2cc95d095ca1456fb524fd464732f8b9"
format_stamp: "Formatted at 2025-01-12 20:51:29 on dist-test-slave-3gql"
I20250112 20:51:29.776207 2666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "2cc95d095ca1456fb524fd464732f8b9"
format_stamp: "Formatted at 2025-01-12 20:51:29 on dist-test-slave-3gql"
I20250112 20:51:29.782137 2666 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250112 20:51:29.785573 2680 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:29.786950 2666 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250112 20:51:29.787142 2666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "2cc95d095ca1456fb524fd464732f8b9"
format_stamp: "Formatted at 2025-01-12 20:51:29 on dist-test-slave-3gql"
I20250112 20:51:29.787374 2666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:29.825176 2666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:29.826030 2666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:29.826294 2666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:29.828991 2666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:29.829090 2666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:29.829290 2666 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:29.829347 2666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:29.882228 2666 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45385
I20250112 20:51:29.882259 2799 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45385 every 8 connection(s)
I20250112 20:51:29.883639 2666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:29.885874 2800 heartbeater.cc:346] Connected to a master server at 127.0.104.62:45417
I20250112 20:51:29.886098 2800 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:29.886978 2800 heartbeater.cc:510] Master 127.0.104.62:45417 requested a full tablet report, sending...
I20250112 20:51:29.887080 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 2666
I20250112 20:51:29.887290 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:29.889263 2592 ts_manager.cc:194] Registered new tserver with Master: 2cc95d095ca1456fb524fd464732f8b9 (127.0.104.1:45385)
I20250112 20:51:29.890897 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45417
--builtin_ntp_servers=127.0.104.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:29.891233 2592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:41775
W20250112 20:51:30.217886 2802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:30.218206 2802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:30.218309 2802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:30.226215 2802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:30.226363 2802 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:30.226469 2802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:30.235879 2802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:45417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:30.237475 2802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:30.239218 2802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:30.250970 2808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:30.251029 2810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:30.251565 2802 server_base.cc:1034] running on GCE node
W20250112 20:51:30.251655 2807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:30.252285 2802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:30.253199 2802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:30.254403 2802 hybrid_clock.cc:648] HybridClock initialized: now 1736715090254341 us; error 53 us; skew 500 ppm
I20250112 20:51:30.254695 2802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:30.256994 2802 webserver.cc:458] Webserver started at http://127.0.104.2:37835/ using document root <none> and password file <none>
I20250112 20:51:30.257791 2802 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:30.257887 2802 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:30.258141 2802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:30.260615 2802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "cb82a16fb2d2424e989fc6a02c936f37"
format_stamp: "Formatted at 2025-01-12 20:51:30 on dist-test-slave-3gql"
I20250112 20:51:30.261353 2802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "cb82a16fb2d2424e989fc6a02c936f37"
format_stamp: "Formatted at 2025-01-12 20:51:30 on dist-test-slave-3gql"
I20250112 20:51:30.266214 2802 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:30.269402 2816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:30.270633 2802 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250112 20:51:30.270768 2802 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "cb82a16fb2d2424e989fc6a02c936f37"
format_stamp: "Formatted at 2025-01-12 20:51:30 on dist-test-slave-3gql"
I20250112 20:51:30.270958 2802 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:30.335116 2802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:30.335968 2802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:30.336231 2802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:30.338917 2802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:30.339017 2802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:30.339147 2802 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:30.339226 2802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:30.381111 2802 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33015
I20250112 20:51:30.381143 2935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33015 every 8 connection(s)
I20250112 20:51:30.382463 2802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:30.384474 2936 heartbeater.cc:346] Connected to a master server at 127.0.104.62:45417
I20250112 20:51:30.384747 2936 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:30.385468 2936 heartbeater.cc:510] Master 127.0.104.62:45417 requested a full tablet report, sending...
I20250112 20:51:30.386746 2592 ts_manager.cc:194] Registered new tserver with Master: cb82a16fb2d2424e989fc6a02c936f37 (127.0.104.2:33015)
I20250112 20:51:30.387578 2592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:54699
I20250112 20:51:30.392004 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 2802
I20250112 20:51:30.392184 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:30.395131 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45417
--builtin_ntp_servers=127.0.104.20:44505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:30.416685 2592 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45178:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:30.717425 2938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:30.717767 2938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:30.717923 2938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:30.726106 2938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:30.726305 2938 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:30.726445 2938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:30.735683 2938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:45417
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:30.737425 2938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:30.739068 2938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:30.750588 2951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:30.750615 2948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:30.750545 2949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:30.751413 2938 server_base.cc:1034] running on GCE node
I20250112 20:51:30.751996 2938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:30.752964 2938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:30.754161 2938 hybrid_clock.cc:648] HybridClock initialized: now 1736715090754118 us; error 33 us; skew 500 ppm
I20250112 20:51:30.754463 2938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:30.756659 2938 webserver.cc:458] Webserver started at http://127.0.104.3:36947/ using document root <none> and password file <none>
I20250112 20:51:30.757468 2938 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:30.757594 2938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:30.757982 2938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:30.760607 2938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "3eb5bf35c5b942aa870b2ab08e194dd5"
format_stamp: "Formatted at 2025-01-12 20:51:30 on dist-test-slave-3gql"
I20250112 20:51:30.761365 2938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "3eb5bf35c5b942aa870b2ab08e194dd5"
format_stamp: "Formatted at 2025-01-12 20:51:30 on dist-test-slave-3gql"
I20250112 20:51:30.766532 2938 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:30.769891 2957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:30.771309 2938 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250112 20:51:30.771449 2938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "3eb5bf35c5b942aa870b2ab08e194dd5"
format_stamp: "Formatted at 2025-01-12 20:51:30 on dist-test-slave-3gql"
I20250112 20:51:30.771636 2938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:30.807255 2938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:30.808121 2938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:30.808398 2938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:30.811125 2938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:30.811213 2938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:30.811339 2938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:30.811398 2938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:30.853117 2938 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36383
I20250112 20:51:30.853147 3076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36383 every 8 connection(s)
I20250112 20:51:30.854576 2938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:30.856853 3077 heartbeater.cc:346] Connected to a master server at 127.0.104.62:45417
I20250112 20:51:30.857080 3077 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:30.857806 3077 heartbeater.cc:510] Master 127.0.104.62:45417 requested a full tablet report, sending...
I20250112 20:51:30.859313 2592 ts_manager.cc:194] Registered new tserver with Master: 3eb5bf35c5b942aa870b2ab08e194dd5 (127.0.104.3:36383)
I20250112 20:51:30.860108 2592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:45411
I20250112 20:51:30.863760 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 2938
I20250112 20:51:30.863895 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:30.866901 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:30.881038 2592 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45194:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:30.881976 2592 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:30.894765 2800 heartbeater.cc:502] Master 127.0.104.62:45417 was elected leader, sending a full tablet report...
I20250112 20:51:30.908152 3007 tablet_service.cc:1467] Processing CreateTablet for tablet b46f38e0572c4e9cb1f3c588c9aee853 (DEFAULT_TABLE table=txn_write_ops_test [id=746c4b3c93f04816b62d4199fbc756b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.908152 2866 tablet_service.cc:1467] Processing CreateTablet for tablet b731368756f6404184ef8c3fa53e7787 (DEFAULT_TABLE table=txn_write_ops_test [id=746c4b3c93f04816b62d4199fbc756b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.909755 3007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46f38e0572c4e9cb1f3c588c9aee853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.909835 2868 tablet_service.cc:1467] Processing CreateTablet for tablet b46f38e0572c4e9cb1f3c588c9aee853 (DEFAULT_TABLE table=txn_write_ops_test [id=746c4b3c93f04816b62d4199fbc756b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.910758 2868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46f38e0572c4e9cb1f3c588c9aee853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.911536 2735 tablet_service.cc:1467] Processing CreateTablet for tablet b46f38e0572c4e9cb1f3c588c9aee853 (DEFAULT_TABLE table=txn_write_ops_test [id=746c4b3c93f04816b62d4199fbc756b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.912765 2735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b46f38e0572c4e9cb1f3c588c9aee853. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.913551 2734 tablet_service.cc:1467] Processing CreateTablet for tablet b731368756f6404184ef8c3fa53e7787 (DEFAULT_TABLE table=txn_write_ops_test [id=746c4b3c93f04816b62d4199fbc756b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.913653 3006 tablet_service.cc:1467] Processing CreateTablet for tablet b731368756f6404184ef8c3fa53e7787 (DEFAULT_TABLE table=txn_write_ops_test [id=746c4b3c93f04816b62d4199fbc756b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:30.913955 2734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b731368756f6404184ef8c3fa53e7787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.914106 3006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b731368756f6404184ef8c3fa53e7787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.917013 2866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b731368756f6404184ef8c3fa53e7787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:30.930210 3095 tablet_bootstrap.cc:492] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37: Bootstrap starting.
I20250112 20:51:30.934376 3095 tablet_bootstrap.cc:654] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.935964 3095 log.cc:826] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:30.936301 3096 tablet_bootstrap.cc:492] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5: Bootstrap starting.
I20250112 20:51:30.937273 3094 tablet_bootstrap.cc:492] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9: Bootstrap starting.
I20250112 20:51:30.940104 3096 tablet_bootstrap.cc:654] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.940737 3094 tablet_bootstrap.cc:654] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.941470 3096 log.cc:826] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:30.941929 3094 log.cc:826] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:30.943969 3096 tablet_bootstrap.cc:492] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5: No bootstrap required, opened a new log
I20250112 20:51:30.944233 3095 tablet_bootstrap.cc:492] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37: No bootstrap required, opened a new log
I20250112 20:51:30.944249 3096 ts_tablet_manager.cc:1397] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250112 20:51:30.944478 3095 ts_tablet_manager.cc:1397] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.007s
I20250112 20:51:30.949530 3094 tablet_bootstrap.cc:492] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9: No bootstrap required, opened a new log
I20250112 20:51:30.949808 3094 ts_tablet_manager.cc:1397] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20250112 20:51:30.951206 3096 raft_consensus.cc:357] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:30.951552 3096 raft_consensus.cc:383] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.951658 3096 raft_consensus.cc:738] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb5bf35c5b942aa870b2ab08e194dd5, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.951548 3095 raft_consensus.cc:357] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:30.951951 3095 raft_consensus.cc:383] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.952039 3095 raft_consensus.cc:738] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb82a16fb2d2424e989fc6a02c936f37, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.952304 3096 consensus_queue.cc:260] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:30.952658 3095 consensus_queue.cc:260] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:30.953544 3077 heartbeater.cc:502] Master 127.0.104.62:45417 was elected leader, sending a full tablet report...
I20250112 20:51:30.954134 3096 ts_tablet_manager.cc:1428] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20250112 20:51:30.954607 3096 tablet_bootstrap.cc:492] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5: Bootstrap starting.
I20250112 20:51:30.956427 3096 tablet_bootstrap.cc:654] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.960635 3095 ts_tablet_manager.cc:1428] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20250112 20:51:30.960525 3094 raft_consensus.cc:357] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:30.960985 2936 heartbeater.cc:502] Master 127.0.104.62:45417 was elected leader, sending a full tablet report...
I20250112 20:51:30.961000 3094 raft_consensus.cc:383] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.961089 3094 raft_consensus.cc:738] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc95d095ca1456fb524fd464732f8b9, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.961967 3094 consensus_queue.cc:260] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:30.964888 3095 tablet_bootstrap.cc:492] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37: Bootstrap starting.
I20250112 20:51:30.967525 3095 tablet_bootstrap.cc:654] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.969838 3094 ts_tablet_manager.cc:1428] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
I20250112 20:51:30.970043 3096 tablet_bootstrap.cc:492] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5: No bootstrap required, opened a new log
I20250112 20:51:30.970193 3096 ts_tablet_manager.cc:1397] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20250112 20:51:30.970407 3094 tablet_bootstrap.cc:492] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9: Bootstrap starting.
I20250112 20:51:30.970929 3096 raft_consensus.cc:357] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:30.971149 3096 raft_consensus.cc:383] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.971246 3096 raft_consensus.cc:738] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb5bf35c5b942aa870b2ab08e194dd5, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.971487 3096 consensus_queue.cc:260] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:30.971822 3095 tablet_bootstrap.cc:492] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37: No bootstrap required, opened a new log
I20250112 20:51:30.971953 3095 ts_tablet_manager.cc:1397] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250112 20:51:30.971961 3096 ts_tablet_manager.cc:1428] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:30.972231 3094 tablet_bootstrap.cc:654] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:30.972851 3095 raft_consensus.cc:357] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:30.973042 3095 raft_consensus.cc:383] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.973136 3095 raft_consensus.cc:738] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb82a16fb2d2424e989fc6a02c936f37, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.973441 3095 consensus_queue.cc:260] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:30.976172 3094 tablet_bootstrap.cc:492] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9: No bootstrap required, opened a new log
I20250112 20:51:30.976284 3095 ts_tablet_manager.cc:1428] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:30.976310 3094 ts_tablet_manager.cc:1397] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250112 20:51:30.977046 3094 raft_consensus.cc:357] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:30.977221 3094 raft_consensus.cc:383] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:30.977334 3094 raft_consensus.cc:738] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc95d095ca1456fb524fd464732f8b9, State: Initialized, Role: FOLLOWER
I20250112 20:51:30.977519 3094 consensus_queue.cc:260] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:30.977984 3094 ts_tablet_manager.cc:1428] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250112 20:51:31.106302 3078 tablet.cc:2367] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:31.134438 2937 tablet.cc:2367] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:31.136057 2801 tablet.cc:2367] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:31.193300 3103 raft_consensus.cc:491] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.193603 3103 raft_consensus.cc:513] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:31.195271 3103 leader_election.cc:290] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eb5bf35c5b942aa870b2ab08e194dd5 (127.0.104.3:36383), cb82a16fb2d2424e989fc6a02c936f37 (127.0.104.2:33015)
I20250112 20:51:31.202160 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46f38e0572c4e9cb1f3c588c9aee853" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb82a16fb2d2424e989fc6a02c936f37" is_pre_election: true
I20250112 20:51:31.202311 3032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46f38e0572c4e9cb1f3c588c9aee853" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" is_pre_election: true
I20250112 20:51:31.202788 3032 raft_consensus.cc:2463] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 0.
I20250112 20:51:31.202788 2891 raft_consensus.cc:2463] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 0.
I20250112 20:51:31.203501 2681 leader_election.cc:304] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc95d095ca1456fb524fd464732f8b9, 3eb5bf35c5b942aa870b2ab08e194dd5; no voters:
I20250112 20:51:31.203881 3103 raft_consensus.cc:2798] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.204027 3103 raft_consensus.cc:491] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.204115 3103 raft_consensus.cc:3054] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.206283 3103 raft_consensus.cc:513] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:31.206851 3103 leader_election.cc:290] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 election: Requested vote from peers 3eb5bf35c5b942aa870b2ab08e194dd5 (127.0.104.3:36383), cb82a16fb2d2424e989fc6a02c936f37 (127.0.104.2:33015)
I20250112 20:51:31.207332 3032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46f38e0572c4e9cb1f3c588c9aee853" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5"
I20250112 20:51:31.207332 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b46f38e0572c4e9cb1f3c588c9aee853" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb82a16fb2d2424e989fc6a02c936f37"
I20250112 20:51:31.207552 2891 raft_consensus.cc:3054] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.207552 3032 raft_consensus.cc:3054] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.210669 3032 raft_consensus.cc:2463] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 1.
I20250112 20:51:31.210655 2891 raft_consensus.cc:2463] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 1.
I20250112 20:51:31.211071 2681 leader_election.cc:304] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2cc95d095ca1456fb524fd464732f8b9, 3eb5bf35c5b942aa870b2ab08e194dd5, cb82a16fb2d2424e989fc6a02c936f37; no voters:
I20250112 20:51:31.211406 3103 raft_consensus.cc:2798] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.211828 3103 raft_consensus.cc:695] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc95d095ca1456fb524fd464732f8b9, State: Running, Role: LEADER
I20250112 20:51:31.212364 3103 consensus_queue.cc:237] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } }
I20250112 20:51:31.216694 2591 catalog_manager.cc:5526] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc95d095ca1456fb524fd464732f8b9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:31.252441 3101 raft_consensus.cc:491] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.252704 3101 raft_consensus.cc:513] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.254441 3101 leader_election.cc:290] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cc95d095ca1456fb524fd464732f8b9 (127.0.104.1:45385), 3eb5bf35c5b942aa870b2ab08e194dd5 (127.0.104.3:36383)
I20250112 20:51:31.261749 2755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b731368756f6404184ef8c3fa53e7787" candidate_uuid: "cb82a16fb2d2424e989fc6a02c936f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc95d095ca1456fb524fd464732f8b9" is_pre_election: true
I20250112 20:51:31.262069 3032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b731368756f6404184ef8c3fa53e7787" candidate_uuid: "cb82a16fb2d2424e989fc6a02c936f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" is_pre_election: true
I20250112 20:51:31.262188 2755 raft_consensus.cc:2463] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb82a16fb2d2424e989fc6a02c936f37 in term 0.
I20250112 20:51:31.262333 3032 raft_consensus.cc:2463] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb82a16fb2d2424e989fc6a02c936f37 in term 0.
I20250112 20:51:31.262750 2817 leader_election.cc:304] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3eb5bf35c5b942aa870b2ab08e194dd5, cb82a16fb2d2424e989fc6a02c936f37; no voters:
I20250112 20:51:31.263140 3101 raft_consensus.cc:2798] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.263257 3101 raft_consensus.cc:491] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.263366 3101 raft_consensus.cc:3054] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.265268 3101 raft_consensus.cc:513] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.265805 3101 leader_election.cc:290] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [CANDIDATE]: Term 1 election: Requested vote from peers 2cc95d095ca1456fb524fd464732f8b9 (127.0.104.1:45385), 3eb5bf35c5b942aa870b2ab08e194dd5 (127.0.104.3:36383)
I20250112 20:51:31.266263 2755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b731368756f6404184ef8c3fa53e7787" candidate_uuid: "cb82a16fb2d2424e989fc6a02c936f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc95d095ca1456fb524fd464732f8b9"
I20250112 20:51:31.266299 3032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b731368756f6404184ef8c3fa53e7787" candidate_uuid: "cb82a16fb2d2424e989fc6a02c936f37" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5"
I20250112 20:51:31.266481 2755 raft_consensus.cc:3054] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.266492 3032 raft_consensus.cc:3054] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.269493 2755 raft_consensus.cc:2463] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb82a16fb2d2424e989fc6a02c936f37 in term 1.
I20250112 20:51:31.269572 3032 raft_consensus.cc:2463] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb82a16fb2d2424e989fc6a02c936f37 in term 1.
I20250112 20:51:31.269984 2817 leader_election.cc:304] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc95d095ca1456fb524fd464732f8b9, cb82a16fb2d2424e989fc6a02c936f37; no voters:
I20250112 20:51:31.270296 3101 raft_consensus.cc:2798] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.270867 3101 raft_consensus.cc:695] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [term 1 LEADER]: Becoming Leader. State: Replica: cb82a16fb2d2424e989fc6a02c936f37, State: Running, Role: LEADER
I20250112 20:51:31.271391 3101 consensus_queue.cc:237] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.275840 2591 catalog_manager.cc:5526] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb82a16fb2d2424e989fc6a02c936f37 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "cb82a16fb2d2424e989fc6a02c936f37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:31.308888 3081 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:51:31.433290 2591 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45210:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:31.434214 2591 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:31.441015 2866 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2ef5ecf7ce41f58953819d953459d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77ca3eeeaf8542789325c8b111252b1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:31.440958 2735 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2ef5ecf7ce41f58953819d953459d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77ca3eeeaf8542789325c8b111252b1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:31.441995 2735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2ef5ecf7ce41f58953819d953459d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.441988 2866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2ef5ecf7ce41f58953819d953459d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.449244 3095 tablet_bootstrap.cc:492] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: Bootstrap starting.
I20250112 20:51:31.449107 3006 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2ef5ecf7ce41f58953819d953459d4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77ca3eeeaf8542789325c8b111252b1f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:31.449972 3006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2ef5ecf7ce41f58953819d953459d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:31.451725 3095 tablet_bootstrap.cc:654] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.454015 3095 tablet_bootstrap.cc:492] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: No bootstrap required, opened a new log
I20250112 20:51:31.454216 3095 ts_tablet_manager.cc:1397] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:31.455035 3096 tablet_bootstrap.cc:492] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: Bootstrap starting.
I20250112 20:51:31.455471 3095 raft_consensus.cc:357] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.455862 3095 raft_consensus.cc:383] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.455956 3095 raft_consensus.cc:738] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb82a16fb2d2424e989fc6a02c936f37, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.456328 3094 tablet_bootstrap.cc:492] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: Bootstrap starting.
I20250112 20:51:31.456482 3095 consensus_queue.cc:260] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.456815 3096 tablet_bootstrap.cc:654] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.457158 3095 ts_tablet_manager.cc:1428] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:31.457082 3101 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.457345 3101 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:31.458197 3094 tablet_bootstrap.cc:654] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:31.458786 3096 tablet_bootstrap.cc:492] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: No bootstrap required, opened a new log
I20250112 20:51:31.458961 3096 ts_tablet_manager.cc:1397] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250112 20:51:31.459717 3096 raft_consensus.cc:357] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.459985 3096 raft_consensus.cc:383] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.460065 3096 raft_consensus.cc:738] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb5bf35c5b942aa870b2ab08e194dd5, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.460086 3094 tablet_bootstrap.cc:492] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: No bootstrap required, opened a new log
I20250112 20:51:31.460250 3094 ts_tablet_manager.cc:1397] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:31.460371 3096 consensus_queue.cc:260] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.460911 3096 ts_tablet_manager.cc:1428] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:31.461268 3094 raft_consensus.cc:357] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.461308 3100 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.461555 3100 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:31.461551 3094 raft_consensus.cc:383] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:31.461640 3094 raft_consensus.cc:738] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc95d095ca1456fb524fd464732f8b9, State: Initialized, Role: FOLLOWER
I20250112 20:51:31.462023 3094 consensus_queue.cc:260] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.462692 3094 ts_tablet_manager.cc:1428] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:31.462551 3103 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.462844 3103 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:31.472091 3103 raft_consensus.cc:491] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:31.472290 3103 raft_consensus.cc:513] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.473066 3103 leader_election.cc:290] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb82a16fb2d2424e989fc6a02c936f37 (127.0.104.2:33015), 3eb5bf35c5b942aa870b2ab08e194dd5 (127.0.104.3:36383)
I20250112 20:51:31.473493 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ef5ecf7ce41f58953819d953459d4" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb82a16fb2d2424e989fc6a02c936f37" is_pre_election: true
I20250112 20:51:31.473623 3032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ef5ecf7ce41f58953819d953459d4" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" is_pre_election: true
I20250112 20:51:31.473804 2891 raft_consensus.cc:2463] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 0.
I20250112 20:51:31.473873 3032 raft_consensus.cc:2463] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 0.
I20250112 20:51:31.474215 2682 leader_election.cc:304] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc95d095ca1456fb524fd464732f8b9, cb82a16fb2d2424e989fc6a02c936f37; no voters:
I20250112 20:51:31.474577 3103 raft_consensus.cc:2798] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:31.474684 3103 raft_consensus.cc:491] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:31.474817 3103 raft_consensus.cc:3054] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.476794 3103 raft_consensus.cc:513] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.477500 3103 leader_election.cc:290] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 election: Requested vote from peers cb82a16fb2d2424e989fc6a02c936f37 (127.0.104.2:33015), 3eb5bf35c5b942aa870b2ab08e194dd5 (127.0.104.3:36383)
I20250112 20:51:31.477877 2891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ef5ecf7ce41f58953819d953459d4" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb82a16fb2d2424e989fc6a02c936f37"
I20250112 20:51:31.477924 3032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e2ef5ecf7ce41f58953819d953459d4" candidate_uuid: "2cc95d095ca1456fb524fd464732f8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5"
I20250112 20:51:31.478086 2891 raft_consensus.cc:3054] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.478116 3032 raft_consensus.cc:3054] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:31.479969 2891 raft_consensus.cc:2463] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 1.
I20250112 20:51:31.479980 3032 raft_consensus.cc:2463] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cc95d095ca1456fb524fd464732f8b9 in term 1.
I20250112 20:51:31.480336 2681 leader_election.cc:304] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2cc95d095ca1456fb524fd464732f8b9, 3eb5bf35c5b942aa870b2ab08e194dd5, cb82a16fb2d2424e989fc6a02c936f37; no voters:
I20250112 20:51:31.480607 3103 raft_consensus.cc:2798] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:31.480772 3103 raft_consensus.cc:695] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 2cc95d095ca1456fb524fd464732f8b9, State: Running, Role: LEADER
I20250112 20:51:31.481006 3103 consensus_queue.cc:237] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } }
I20250112 20:51:31.482483 3108 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: TxnStatusTablet state changed. Reason: New leader 2cc95d095ca1456fb524fd464732f8b9. Latest consensus state: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.482843 3108 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:31.483419 3125 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:31.483776 3125 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:31.483879 2591 catalog_manager.cc:5526] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cc95d095ca1456fb524fd464732f8b9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:31.529974 3032 raft_consensus.cc:1270] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 1 FOLLOWER]: Refusing update from remote peer 2cc95d095ca1456fb524fd464732f8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:31.530009 2891 raft_consensus.cc:1270] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37 [term 1 FOLLOWER]: Refusing update from remote peer 2cc95d095ca1456fb524fd464732f8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:31.530942 3108 consensus_queue.cc:1035] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.531293 3103 consensus_queue.cc:1035] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.541777 3100 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: TxnStatusTablet state changed. Reason: New leader 2cc95d095ca1456fb524fd464732f8b9. Latest consensus state: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.542027 3100 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:31.543609 3101 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: TxnStatusTablet state changed. Reason: New leader 2cc95d095ca1456fb524fd464732f8b9. Latest consensus state: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.543941 3101 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:31.546945 3129 mvcc.cc:204] Tried to move back new op lower bound from 7113585014895890432 to 7113585014707986432. Current Snapshot: MvccSnapshot[applied={T|T < 7113585014895890432}]
I20250112 20:51:31.555063 3100 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.555286 3100 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P 3eb5bf35c5b942aa870b2ab08e194dd5: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:31.560042 3101 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.560320 3101 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P cb82a16fb2d2424e989fc6a02c936f37: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:31.563184 3108 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.563436 3108 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:31.563643 3103 tablet_replica.cc:440] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cc95d095ca1456fb524fd464732f8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 } } peers { permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 } } peers { permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 } } }
I20250112 20:51:31.563894 3103 tablet_replica.cc:443] T 6e2ef5ecf7ce41f58953819d953459d4 P 2cc95d095ca1456fb524fd464732f8b9: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:31.597936 3032 raft_consensus.cc:1270] T b46f38e0572c4e9cb1f3c588c9aee853 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 1 FOLLOWER]: Refusing update from remote peer 2cc95d095ca1456fb524fd464732f8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:31.597909 2891 raft_consensus.cc:1270] T b46f38e0572c4e9cb1f3c588c9aee853 P cb82a16fb2d2424e989fc6a02c936f37 [term 1 FOLLOWER]: Refusing update from remote peer 2cc95d095ca1456fb524fd464732f8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:31.598448 3136 consensus_queue.cc:1035] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.598713 3103 consensus_queue.cc:1035] T b46f38e0572c4e9cb1f3c588c9aee853 P 2cc95d095ca1456fb524fd464732f8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb82a16fb2d2424e989fc6a02c936f37" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.603715 3128 mvcc.cc:204] Tried to move back new op lower bound from 7113585015180169216 to 7113585013609492480. Current Snapshot: MvccSnapshot[applied={T|T < 7113585015180169216}]
I20250112 20:51:31.648864 3032 raft_consensus.cc:1270] T b731368756f6404184ef8c3fa53e7787 P 3eb5bf35c5b942aa870b2ab08e194dd5 [term 1 FOLLOWER]: Refusing update from remote peer cb82a16fb2d2424e989fc6a02c936f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:31.649075 2755 raft_consensus.cc:1270] T b731368756f6404184ef8c3fa53e7787 P 2cc95d095ca1456fb524fd464732f8b9 [term 1 FOLLOWER]: Refusing update from remote peer cb82a16fb2d2424e989fc6a02c936f37: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:31.649824 3101 consensus_queue.cc:1035] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb5bf35c5b942aa870b2ab08e194dd5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.650108 3112 consensus_queue.cc:1035] T b731368756f6404184ef8c3fa53e7787 P cb82a16fb2d2424e989fc6a02c936f37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc95d095ca1456fb524fd464732f8b9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:31.655436 3130 mvcc.cc:204] Tried to move back new op lower bound from 7113585015387389952 to 7113585013851643904. Current Snapshot: MvccSnapshot[applied={T|T < 7113585015387389952}]
I20250112 20:51:36.748369 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 2666
I20250112 20:51:36.749464 2795 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:36.901300 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 2666
I20250112 20:51:36.937651 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 2802
I20250112 20:51:36.938721 2931 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:37.083810 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 2802
I20250112 20:51:37.106818 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 2938
I20250112 20:51:37.107892 3072 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:37.220773 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 2938
I20250112 20:51:37.241127 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 2561
I20250112 20:51:37.242274 2632 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:37.347977 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 2561
2025-01-12T20:51:37Z chronyd exiting
[ OK ] TxnWriteOpsITest.TxnMultipleSingleRowWritesCommit (8578 ms)
[ RUN ] TxnWriteOpsITest.FrequentElections
2025-01-12T20:51:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:37Z Disabled control of system clock
I20250112 20:51:37.415098 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44273
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44273
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20250112 20:51:37.726472 3181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:37.726850 3181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:37.727002 3181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:37.735594 3181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:37.735726 3181 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:37.735769 3181 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20250112 20:51:37.735831 3181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:37.735883 3181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:37.735949 3181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:37.745584 3181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44273
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:37.747223 3181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:37.748878 3181 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:37.758728 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:37.758662 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:37.758702 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:37.759521 3181 server_base.cc:1034] running on GCE node
I20250112 20:51:37.760213 3181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:37.761552 3181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:37.762764 3181 hybrid_clock.cc:648] HybridClock initialized: now 1736715097762709 us; error 30 us; skew 500 ppm
I20250112 20:51:37.763101 3181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:37.769271 3181 webserver.cc:458] Webserver started at http://127.0.104.62:45001/ using document root <none> and password file <none>
I20250112 20:51:37.770023 3181 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:37.770152 3181 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:37.770476 3181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:37.773167 3181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "993aca280e744b2a9f3ddf4fc46222f5"
format_stamp: "Formatted at 2025-01-12 20:51:37 on dist-test-slave-3gql"
I20250112 20:51:37.773919 3181 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "993aca280e744b2a9f3ddf4fc46222f5"
format_stamp: "Formatted at 2025-01-12 20:51:37 on dist-test-slave-3gql"
I20250112 20:51:37.779100 3181 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:37.782279 3195 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:37.783635 3181 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250112 20:51:37.783785 3181 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "993aca280e744b2a9f3ddf4fc46222f5"
format_stamp: "Formatted at 2025-01-12 20:51:37 on dist-test-slave-3gql"
I20250112 20:51:37.783954 3181 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:37.803426 3181 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:37.804265 3181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:37.804597 3181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:37.823562 3181 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44273
I20250112 20:51:37.823606 3256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44273 every 8 connection(s)
I20250112 20:51:37.824954 3181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:37.828811 3257 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:37.832617 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 3181
I20250112 20:51:37.832886 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:37.839501 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5: Bootstrap starting.
I20250112 20:51:37.843310 3257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:37.844508 3257 log.cc:826] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:37.847123 3257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5: No bootstrap required, opened a new log
I20250112 20:51:37.853030 3257 raft_consensus.cc:357] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993aca280e744b2a9f3ddf4fc46222f5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44273 } }
I20250112 20:51:37.853366 3257 raft_consensus.cc:383] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:37.853494 3257 raft_consensus.cc:738] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 993aca280e744b2a9f3ddf4fc46222f5, State: Initialized, Role: FOLLOWER
I20250112 20:51:37.854036 3257 consensus_queue.cc:260] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993aca280e744b2a9f3ddf4fc46222f5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44273 } }
I20250112 20:51:37.854225 3257 raft_consensus.cc:397] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:37.854347 3257 raft_consensus.cc:491] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:37.854491 3257 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:37.856406 3257 raft_consensus.cc:513] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993aca280e744b2a9f3ddf4fc46222f5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44273 } }
I20250112 20:51:37.856860 3257 leader_election.cc:304] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 993aca280e744b2a9f3ddf4fc46222f5; no voters:
I20250112 20:51:37.857391 3257 leader_election.cc:290] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:37.857518 3262 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:37.858130 3262 raft_consensus.cc:695] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [term 1 LEADER]: Becoming Leader. State: Replica: 993aca280e744b2a9f3ddf4fc46222f5, State: Running, Role: LEADER
I20250112 20:51:37.858615 3262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993aca280e744b2a9f3ddf4fc46222f5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44273 } }
I20250112 20:51:37.859256 3257 sys_catalog.cc:564] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:37.861539 3263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "993aca280e744b2a9f3ddf4fc46222f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993aca280e744b2a9f3ddf4fc46222f5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44273 } } }
I20250112 20:51:37.862111 3263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:37.861547 3264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 993aca280e744b2a9f3ddf4fc46222f5. Latest consensus state: current_term: 1 leader_uuid: "993aca280e744b2a9f3ddf4fc46222f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "993aca280e744b2a9f3ddf4fc46222f5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44273 } } }
I20250112 20:51:37.863687 3264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:37.865394 3269 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:37.870674 3269 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:37.879737 3269 catalog_manager.cc:1348] Generated new cluster ID: a4b469a63fec4245ab041afb704807bb
I20250112 20:51:37.879853 3269 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:37.907383 3269 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:37.909054 3269 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:37.919111 3269 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 993aca280e744b2a9f3ddf4fc46222f5: Generated new TSK 0
I20250112 20:51:37.920078 3269 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:37.927085 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44273
--builtin_ntp_servers=127.0.104.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--heartbeat_interval_ms=64
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20250112 20:51:37.928325 3212 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57808:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:37.938010 3257 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20250112 20:51:38.283104 3286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:38.283382 3286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:38.283470 3286 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20250112 20:51:38.283540 3286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:38.291208 3286 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20250112 20:51:38.291337 3286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:38.291532 3286 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:38.291658 3286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:38.300866 3286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true
--heartbeat_interval_ms=64
--tserver_master_addrs=127.0.104.62:44273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:38.302510 3286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:38.304198 3286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:38.315529 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:38.315475 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:38.315673 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:38.316284 3286 server_base.cc:1034] running on GCE node
I20250112 20:51:38.316907 3286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:38.317752 3286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:38.318934 3286 hybrid_clock.cc:648] HybridClock initialized: now 1736715098318894 us; error 31 us; skew 500 ppm
I20250112 20:51:38.319211 3286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:38.321391 3286 webserver.cc:458] Webserver started at http://127.0.104.1:33575/ using document root <none> and password file <none>
I20250112 20:51:38.322091 3286 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:38.322180 3286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:38.322481 3286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:38.324977 3286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
format_stamp: "Formatted at 2025-01-12 20:51:38 on dist-test-slave-3gql"
I20250112 20:51:38.325659 3286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
format_stamp: "Formatted at 2025-01-12 20:51:38 on dist-test-slave-3gql"
I20250112 20:51:38.330543 3286 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:38.333660 3300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:38.334905 3286 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:38.335053 3286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
format_stamp: "Formatted at 2025-01-12 20:51:38 on dist-test-slave-3gql"
I20250112 20:51:38.335232 3286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:38.349093 3286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:38.349882 3286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:38.350139 3286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:38.352792 3286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:38.352877 3286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:38.353017 3286 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:38.353081 3286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:38.398397 3286 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:42899
I20250112 20:51:38.398437 3419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:42899 every 8 connection(s)
I20250112 20:51:38.399658 3286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:38.401823 3420 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44273
I20250112 20:51:38.402014 3420 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:38.402702 3420 heartbeater.cc:510] Master 127.0.104.62:44273 requested a full tablet report, sending...
I20250112 20:51:38.404541 3212 ts_manager.cc:194] Registered new tserver with Master: d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:38.406025 3212 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:38633
I20250112 20:51:38.410055 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 3286
I20250112 20:51:38.410225 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:38.413364 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44273
--builtin_ntp_servers=127.0.104.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--heartbeat_interval_ms=64
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20250112 20:51:38.472638 3420 heartbeater.cc:502] Master 127.0.104.62:44273 was elected leader, sending a full tablet report...
W20250112 20:51:38.732079 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:38.732398 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:38.732520 3422 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20250112 20:51:38.732640 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:38.740623 3422 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20250112 20:51:38.740767 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:38.740935 3422 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:38.741055 3422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:38.750066 3422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true
--heartbeat_interval_ms=64
--tserver_master_addrs=127.0.104.62:44273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:38.751710 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:38.753420 3422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:38.765189 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:38.765167 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:38.766062 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:38.766361 3422 server_base.cc:1034] running on GCE node
I20250112 20:51:38.766995 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:38.768018 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:38.769224 3422 hybrid_clock.cc:648] HybridClock initialized: now 1736715098769177 us; error 45 us; skew 500 ppm
I20250112 20:51:38.769536 3422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:38.771865 3422 webserver.cc:458] Webserver started at http://127.0.104.2:38941/ using document root <none> and password file <none>
I20250112 20:51:38.772655 3422 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:38.772774 3422 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:38.773082 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:38.775590 3422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "f15597f529254479b5987d2613686dfb"
format_stamp: "Formatted at 2025-01-12 20:51:38 on dist-test-slave-3gql"
I20250112 20:51:38.776291 3422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "f15597f529254479b5987d2613686dfb"
format_stamp: "Formatted at 2025-01-12 20:51:38 on dist-test-slave-3gql"
I20250112 20:51:38.781402 3422 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250112 20:51:38.784499 3436 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:38.785961 3422 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250112 20:51:38.786129 3422 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "f15597f529254479b5987d2613686dfb"
format_stamp: "Formatted at 2025-01-12 20:51:38 on dist-test-slave-3gql"
I20250112 20:51:38.786346 3422 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:38.809324 3422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:38.810326 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:38.810590 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:38.813346 3422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:38.813436 3422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:38.813575 3422 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:38.813639 3422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:38.859342 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44825
I20250112 20:51:38.859372 3555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44825 every 8 connection(s)
I20250112 20:51:38.860663 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:38.862119 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 3422
I20250112 20:51:38.862349 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:38.862695 3556 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44273
I20250112 20:51:38.862898 3556 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:38.863703 3556 heartbeater.cc:510] Master 127.0.104.62:44273 requested a full tablet report, sending...
I20250112 20:51:38.864923 3212 ts_manager.cc:194] Registered new tserver with Master: f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:38.865533 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44273
--builtin_ntp_servers=127.0.104.20:42603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--heartbeat_interval_ms=64
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20250112 20:51:38.865721 3212 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:55975
I20250112 20:51:38.932154 3556 heartbeater.cc:502] Master 127.0.104.62:44273 was elected leader, sending a full tablet report...
I20250112 20:51:38.951489 3212 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57824:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:39.191620 3558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:39.191924 3558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:39.192054 3558 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20250112 20:51:39.192153 3558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:39.200127 3558 flags.cc:425] Enabled experimental flag: --raft_enable_pre_election=false
W20250112 20:51:39.200255 3558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:39.200434 3558 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:39.200603 3558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:39.209836 3558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_enable_pre_election=false
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true
--heartbeat_interval_ms=64
--tserver_master_addrs=127.0.104.62:44273
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:39.211452 3558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:39.213156 3558 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:39.224227 3569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:39.224212 3571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:39.224220 3568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:39.225133 3558 server_base.cc:1034] running on GCE node
I20250112 20:51:39.225668 3558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:39.226521 3558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:39.227708 3558 hybrid_clock.cc:648] HybridClock initialized: now 1736715099227655 us; error 50 us; skew 500 ppm
I20250112 20:51:39.228005 3558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:39.230226 3558 webserver.cc:458] Webserver started at http://127.0.104.3:40329/ using document root <none> and password file <none>
I20250112 20:51:39.230955 3558 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:39.231062 3558 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:39.231372 3558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:39.234041 3558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "710959530a634cc991544c14efad9e96"
format_stamp: "Formatted at 2025-01-12 20:51:39 on dist-test-slave-3gql"
I20250112 20:51:39.234768 3558 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "710959530a634cc991544c14efad9e96"
format_stamp: "Formatted at 2025-01-12 20:51:39 on dist-test-slave-3gql"
I20250112 20:51:39.239945 3558 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250112 20:51:39.243263 3577 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:39.244635 3558 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250112 20:51:39.244796 3558 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "710959530a634cc991544c14efad9e96"
format_stamp: "Formatted at 2025-01-12 20:51:39 on dist-test-slave-3gql"
I20250112 20:51:39.244987 3558 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:39.280259 3558 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:39.281177 3558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:39.281455 3558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:39.284220 3558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:39.284307 3558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:39.284444 3558 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:39.284533 3558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:39.326678 3558 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44097
I20250112 20:51:39.326733 3696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44097 every 8 connection(s)
I20250112 20:51:39.328083 3558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:39.330255 3697 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44273
I20250112 20:51:39.330502 3697 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:39.331195 3697 heartbeater.cc:510] Master 127.0.104.62:44273 requested a full tablet report, sending...
I20250112 20:51:39.332371 3212 ts_manager.cc:194] Registered new tserver with Master: 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.333225 3212 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:55091
I20250112 20:51:39.334481 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 3558
I20250112 20:51:39.334653 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.FrequentElections.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:39.337725 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:39.352082 3212 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57840:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:39.353061 3212 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.391702 3490 tablet_service.cc:1467] Processing CreateTablet for tablet b8edbeb2c8364a3691b8d3c1289b1e77 (DEFAULT_TABLE table=txn_write_ops_test [id=08d0d85555cf45e1b48dc714d4d8e639]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.391676 3491 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8bedc489994a98b60f7742ba6e3f5c (DEFAULT_TABLE table=txn_write_ops_test [id=08d0d85555cf45e1b48dc714d4d8e639]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.393518 3490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8edbeb2c8364a3691b8d3c1289b1e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.394855 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c8bedc489994a98b60f7742ba6e3f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.395015 3632 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8bedc489994a98b60f7742ba6e3f5c (DEFAULT_TABLE table=txn_write_ops_test [id=08d0d85555cf45e1b48dc714d4d8e639]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.394743 3355 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8bedc489994a98b60f7742ba6e3f5c (DEFAULT_TABLE table=txn_write_ops_test [id=08d0d85555cf45e1b48dc714d4d8e639]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.394774 3631 tablet_service.cc:1467] Processing CreateTablet for tablet b8edbeb2c8364a3691b8d3c1289b1e77 (DEFAULT_TABLE table=txn_write_ops_test [id=08d0d85555cf45e1b48dc714d4d8e639]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.395650 3354 tablet_service.cc:1467] Processing CreateTablet for tablet b8edbeb2c8364a3691b8d3c1289b1e77 (DEFAULT_TABLE table=txn_write_ops_test [id=08d0d85555cf45e1b48dc714d4d8e639]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:39.396149 3355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c8bedc489994a98b60f7742ba6e3f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.396178 3631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8edbeb2c8364a3691b8d3c1289b1e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.396679 3354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8edbeb2c8364a3691b8d3c1289b1e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.399969 3697 heartbeater.cc:502] Master 127.0.104.62:44273 was elected leader, sending a full tablet report...
I20250112 20:51:39.404644 3632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c8bedc489994a98b60f7742ba6e3f5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.415917 3714 tablet_bootstrap.cc:492] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb: Bootstrap starting.
I20250112 20:51:39.419633 3714 tablet_bootstrap.cc:654] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.421643 3714 log.cc:826] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:39.421919 3715 tablet_bootstrap.cc:492] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Bootstrap starting.
I20250112 20:51:39.425617 3716 tablet_bootstrap.cc:492] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Bootstrap starting.
I20250112 20:51:39.426124 3715 tablet_bootstrap.cc:654] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.427377 3715 log.cc:826] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:39.427510 3714 tablet_bootstrap.cc:492] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb: No bootstrap required, opened a new log
I20250112 20:51:39.427781 3714 ts_tablet_manager.cc:1397] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250112 20:51:39.428589 3716 tablet_bootstrap.cc:654] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.429680 3716 log.cc:826] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:39.431229 3715 tablet_bootstrap.cc:492] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: No bootstrap required, opened a new log
I20250112 20:51:39.431509 3715 ts_tablet_manager.cc:1397] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20250112 20:51:39.431875 3716 tablet_bootstrap.cc:492] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: No bootstrap required, opened a new log
I20250112 20:51:39.432090 3716 ts_tablet_manager.cc:1397] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20250112 20:51:39.436537 3714 raft_consensus.cc:357] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.437037 3714 raft_consensus.cc:383] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.437145 3714 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.437872 3714 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.437911 3716 raft_consensus.cc:357] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.438241 3716 raft_consensus.cc:383] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.438328 3716 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.439059 3716 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.439781 3714 ts_tablet_manager.cc:1428] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20250112 20:51:39.440248 3714 tablet_bootstrap.cc:492] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Bootstrap starting.
I20250112 20:51:39.441213 3715 raft_consensus.cc:357] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.441968 3715 raft_consensus.cc:383] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.442117 3715 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.442392 3714 tablet_bootstrap.cc:654] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.442935 3715 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.446729 3715 ts_tablet_manager.cc:1428] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20250112 20:51:39.447127 3715 tablet_bootstrap.cc:492] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Bootstrap starting.
I20250112 20:51:39.447396 3716 ts_tablet_manager.cc:1428] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Time spent starting tablet: real 0.015s user 0.008s sys 0.002s
I20250112 20:51:39.447836 3716 tablet_bootstrap.cc:492] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Bootstrap starting.
I20250112 20:51:39.448925 3715 tablet_bootstrap.cc:654] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.448994 3714 tablet_bootstrap.cc:492] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: No bootstrap required, opened a new log
I20250112 20:51:39.449234 3714 ts_tablet_manager.cc:1397] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250112 20:51:39.449554 3716 tablet_bootstrap.cc:654] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:39.450066 3714 raft_consensus.cc:357] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.450392 3714 raft_consensus.cc:383] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.450502 3714 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.450809 3714 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.451591 3714 ts_tablet_manager.cc:1428] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:39.453437 3716 tablet_bootstrap.cc:492] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: No bootstrap required, opened a new log
I20250112 20:51:39.453480 3715 tablet_bootstrap.cc:492] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: No bootstrap required, opened a new log
I20250112 20:51:39.453604 3716 ts_tablet_manager.cc:1397] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250112 20:51:39.453711 3715 ts_tablet_manager.cc:1397] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250112 20:51:39.454475 3716 raft_consensus.cc:357] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.454536 3715 raft_consensus.cc:357] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.454638 3716 raft_consensus.cc:383] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.454669 3715 raft_consensus.cc:383] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:39.454734 3716 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.454747 3715 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Initialized, Role: FOLLOWER
I20250112 20:51:39.454944 3716 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.454967 3715 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.455420 3716 ts_tablet_manager.cc:1428] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:39.455473 3715 ts_tablet_manager.cc:1428] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:39.516462 3721 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.516743 3721 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.518918 3721 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.520802 3721 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 1 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.528476 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.528605 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.528931 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.529008 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.531096 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 1.
I20250112 20:51:39.531097 3511 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 1.
I20250112 20:51:39.531719 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:39.532141 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.532676 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.533118 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.537974 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:39.580601 3698 tablet.cc:2367] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:39.612645 3557 tablet.cc:2367] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:39.652040 3421 tablet.cc:2367] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:39.774231 3722 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:39.774394 3722 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.776327 3722 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.777917 3722 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 1 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:39.783735 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.784037 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.784621 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.785023 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:39.786231 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 1.
I20250112 20:51:39.786798 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:39.787278 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:39.787294 3375 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 1.
I20250112 20:51:39.787829 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 1 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:39.788321 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.793061 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 0 to 1, leader changed from <none> to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:39.862193 3699 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:51:39.864964 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:39.872583 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:39.872898 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.873032 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.875952 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.876822 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 2 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.877235 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.877339 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.877476 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.877550 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 20:51:39.877627 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:39.877943 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.878118 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:39.878290 3375 raft_consensus.cc:474] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20250112 20:51:39.878408 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.879050 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:39.880380 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 2.
I20250112 20:51:39.880777 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:39.881071 3721 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:51:39.881250 3721 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.881346 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.881549 3721 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.885403 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 1 to 2, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 2 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.886543 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:39.886942 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.887080 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 2 FOLLOWER]: Advancing to term 3
I20250112 20:51:39.889214 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.890568 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 3 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.892031 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:39.892263 3511 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.892390 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.895238 3511 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.896030 3511 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 2 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.896587 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:39.901710 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.901955 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 LEADER]: Stepping down as leader of term 2
I20250112 20:51:39.901842 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.902077 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.902155 3374 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 LEADER]: Stepping down as leader of term 1
I20250112 20:51:39.902292 3374 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.902448 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.902638 3374 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.903056 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 FOLLOWER]: Advancing to term 3
I20250112 20:51:39.903090 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.905743 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.906040 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 2 FOLLOWER]: Advancing to term 3
I20250112 20:51:39.905925 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.906152 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.906169 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:39.909103 3651 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 2.
I20250112 20:51:39.909459 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.909956 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:39.910475 3720 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250112 20:51:39.910912 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:39.911156 3720 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:51:39.911434 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.911631 3720 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 2 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:39.912204 3720 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.917495 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:39.917464 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 1 to 2, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 2 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.917822 3651 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.917944 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 3 FOLLOWER]: Advancing to term 4
I20250112 20:51:39.920648 3651 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.921213 3651 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 4 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:39.921692 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.921932 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 3 FOLLOWER]: Advancing to term 4
I20250112 20:51:39.922072 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.922272 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 3 FOLLOWER]: Advancing to term 4
I20250112 20:51:39.922448 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:39.922622 3651 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.922734 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 2 FOLLOWER]: Advancing to term 3
I20250112 20:51:39.924219 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250112 20:51:39.924710 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 4.
I20250112 20:51:39.925138 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:39.925426 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 20:51:39.925336 3651 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.925601 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 4 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:39.925889 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.926095 3651 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 3 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:39.926523 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.926506 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.926723 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 2 FOLLOWER]: Advancing to term 3
I20250112 20:51:39.927677 3511 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 2 LEADER]: Stepping down as leader of term 2
I20250112 20:51:39.927881 3511 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:39.928261 3511 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.929075 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 2 FOLLOWER]: Advancing to term 3
I20250112 20:51:39.929555 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.930289 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:39.931565 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 2 to 4, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 4 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:39.932138 3511 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 3 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.932722 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:39.932914 3374 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.933040 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 FOLLOWER]: Advancing to term 5
I20250112 20:51:39.935082 3374 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.935717 3374 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 5 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.936141 3578 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:39.936344 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.936506 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250112 20:51:39.936664 3651 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 4 LEADER]: Stepping down as leader of term 4
I20250112 20:51:39.936821 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:39.936815 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.936937 3651 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:39.937016 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 4 FOLLOWER]: Advancing to term 5
I20250112 20:51:39.937011 3374 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.937122 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 3 FOLLOWER]: Advancing to term 4
I20250112 20:51:39.937245 3651 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.937552 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 4 FOLLOWER]: Advancing to term 5
I20250112 20:51:39.938930 3374 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.939589 3374 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 4 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.939945 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 5.
I20250112 20:51:39.940121 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:39.940302 3651 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20250112 20:51:39.940191 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.940397 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:39.940431 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 3 FOLLOWER]: Advancing to term 4
I20250112 20:51:39.940688 3721 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 FOLLOWER]: Leader election won for term 5
I20250112 20:51:39.940836 3721 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.941002 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:39.941140 3721 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.941242 3509 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.941371 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 5 FOLLOWER]: Advancing to term 6
I20250112 20:51:39.942900 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 4 because replica has last-logged OpId of term: 2 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20250112 20:51:39.943516 3509 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.944159 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.944396 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 3 FOLLOWER]: Advancing to term 4
I20250112 20:51:39.944491 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 6 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.944571 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 6 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.944734 3374 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 LEADER]: Stepping down as leader of term 5
I20250112 20:51:39.944748 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 5 FOLLOWER]: Advancing to term 6
I20250112 20:51:39.944842 3374 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.944757 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 4 to 5, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 5 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.945120 3374 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.945403 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 FOLLOWER]: Advancing to term 6
I20250112 20:51:39.947065 3651 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 4.
I20250112 20:51:39.947428 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: f15597f529254479b5987d2613686dfb
I20250112 20:51:39.947597 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.947739 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 FOLLOWER]: Leader election won for term 4
I20250112 20:51:39.947906 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.948125 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.948657 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.948877 3509 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 6 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.949376 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:39.949818 3720 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250112 20:51:39.950099 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:39.950274 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.950397 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 4 FOLLOWER]: Advancing to term 5
I20250112 20:51:39.952191 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 2 to 4, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 4 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.952936 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.953683 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 5 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.954165 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:39.954113 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.954315 3374 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 LEADER]: Stepping down as leader of term 4
I20250112 20:51:39.954259 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.954404 3374 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.954479 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 4 FOLLOWER]: Advancing to term 5
I20250112 20:51:39.954658 3374 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.954941 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 4 FOLLOWER]: Advancing to term 5
I20250112 20:51:39.957058 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20250112 20:51:39.957087 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 5.
I20250112 20:51:39.957489 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:39.957873 3720 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 5 FOLLOWER]: Leader election won for term 5
I20250112 20:51:39.958066 3720 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 5 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:39.958321 3720 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.959793 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:39.960107 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.960317 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 6 FOLLOWER]: Advancing to term 7
I20250112 20:51:39.961162 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 4 to 5, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 5 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.965363 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.968403 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.968734 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 6 FOLLOWER]: Advancing to term 7
I20250112 20:51:39.971383 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 7.
I20250112 20:51:39.971873 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:39.972085 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 7 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:39.972208 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 7 FOLLOWER]: Leader election won for term 7
I20250112 20:51:39.972388 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 7 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:39.972726 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 4.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.972734 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.972990 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 6 FOLLOWER]: Advancing to term 7
I20250112 20:51:39.973526 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:39.973784 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.973948 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 5 FOLLOWER]: Advancing to term 6
I20250112 20:51:39.975054 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 5 to 7, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 7 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:39.975872 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250112 20:51:39.977766 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.978451 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 6 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:39.978943 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 6 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:39.978941 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 6 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.979161 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 5 FOLLOWER]: Advancing to term 6
I20250112 20:51:39.979161 3510 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 5 LEADER]: Stepping down as leader of term 5
I20250112 20:51:39.979244 3510 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:39.979285 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:39.979553 3510 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.979846 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 5 FOLLOWER]: Advancing to term 6
I20250112 20:51:39.980065 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:39.980258 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.980394 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 7 FOLLOWER]: Advancing to term 8
I20250112 20:51:39.983157 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.983038 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.983700 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 6 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:39.983814 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 8 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.984244 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.984337 3578 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:39.984452 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 7 LEADER]: Stepping down as leader of term 7
I20250112 20:51:39.984597 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:39.984704 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20250112 20:51:39.984557 3212 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57852:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:51:39.984862 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.984970 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.985146 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 7 FOLLOWER]: Advancing to term 8
I20250112 20:51:39.985179 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 7 FOLLOWER]: Advancing to term 8
W20250112 20:51:39.985226 3212 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:39.987370 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:39.987975 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 8.
I20250112 20:51:39.988067 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20250112 20:51:39.988363 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:39.988641 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.988770 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 6 FOLLOWER]: Advancing to term 7
I20250112 20:51:39.988991 3728 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Leader election won for term 8
I20250112 20:51:39.989131 3728 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:39.989447 3728 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.991495 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.991943 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 7 to 8, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 8 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:39.992971 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:39.993145 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 6 FOLLOWER]: Advancing to term 7
I20250112 20:51:39.993511 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 7 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:39.994095 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:39.994845 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:39.995075 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:39.995191 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 8 FOLLOWER]: Advancing to term 9
I20250112 20:51:39.994944 3491 tablet_service.cc:1467] Processing CreateTablet for tablet a82c4eb8655241348564eafae773b59d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c37a0a2bbad4da3978c2c2439f984f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:39.995888 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250112 20:51:39.995996 3491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a82c4eb8655241348564eafae773b59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.997252 3355 tablet_service.cc:1467] Processing CreateTablet for tablet a82c4eb8655241348564eafae773b59d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c37a0a2bbad4da3978c2c2439f984f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:39.998028 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:39.998159 3355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a82c4eb8655241348564eafae773b59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.998250 3631 tablet_service.cc:1467] Processing CreateTablet for tablet a82c4eb8655241348564eafae773b59d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5c37a0a2bbad4da3978c2c2439f984f3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:39.999104 3631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a82c4eb8655241348564eafae773b59d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:39.999217 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 9 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:39.999594 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 8 FOLLOWER]: Advancing to term 9
I20250112 20:51:39.999778 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 9 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.000857 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 9 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.001041 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 LEADER]: Stepping down as leader of term 8
I20250112 20:51:40.001152 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.001448 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.002673 3651 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 9 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.003180 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.003393 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.003530 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 7 FOLLOWER]: Advancing to term 8
I20250112 20:51:40.004693 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.005041 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Advancing to term 9
I20250112 20:51:40.005219 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 6 FOLLOWER]: Advancing to term 7
I20250112 20:51:40.006196 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.007330 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.007524 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 7 FOLLOWER]: Advancing to term 8
I20250112 20:51:40.007792 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.008457 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 8 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.009043 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.009814 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.009994 3651 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.010134 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 9 FOLLOWER]: Advancing to term 10
I20250112 20:51:40.010320 3374 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 8.
I20250112 20:51:40.010751 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96
I20250112 20:51:40.011191 3720 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 FOLLOWER]: Leader election won for term 8
I20250112 20:51:40.011343 3720 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.011621 3720 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 5.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.012136 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 9 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.012276 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 7.
I20250112 20:51:40.012933 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: f15597f529254479b5987d2613686dfb
I20250112 20:51:40.012985 3651 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.013319 3728 raft_consensus.cc:2758] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20250112 20:51:40.013330 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.013621 3746 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250112 20:51:40.013935 3651 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 10 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.014156 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.014343 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 9 FOLLOWER]: Advancing to term 10
I20250112 20:51:40.014338 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.014526 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 FOLLOWER]: Advancing to term 10
I20250112 20:51:40.014973 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.015323 3754 tablet_bootstrap.cc:492] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: Bootstrap starting.
I20250112 20:51:40.017184 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 10.
I20250112 20:51:40.017454 3754 tablet_bootstrap.cc:654] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.017697 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.017961 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 10 FOLLOWER]: Leader election won for term 10
I20250112 20:51:40.018139 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 10 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.018442 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.018400 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 7.3, Last appended by leader: 3, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.018643 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.018774 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 7 FOLLOWER]: Advancing to term 8
I20250112 20:51:40.019364 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 10 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20250112 20:51:40.020803 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 8 to 10, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 10 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.021353 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.022017 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 8 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.022481 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 8 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.022496 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 8 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.022593 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.022766 3374 raft_consensus.cc:2388] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 in current term 8: Already voted for candidate f15597f529254479b5987d2613686dfb in this term.
I20250112 20:51:40.022790 3510 raft_consensus.cc:2388] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 LEADER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 in current term 8: Already voted for candidate f15597f529254479b5987d2613686dfb in this term.
I20250112 20:51:40.023360 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.023592 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.023532 3578 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.023730 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 10 FOLLOWER]: Advancing to term 11
I20250112 20:51:40.024055 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20250112 20:51:40.025697 3754 tablet_bootstrap.cc:492] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: No bootstrap required, opened a new log
I20250112 20:51:40.025902 3754 ts_tablet_manager.cc:1397] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20250112 20:51:40.026355 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.026959 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 11 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.027014 3754 raft_consensus.cc:357] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.027680 3754 raft_consensus.cc:383] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.027774 3754 raft_consensus.cc:738] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.027732 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 11 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.027932 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 10 FOLLOWER]: Advancing to term 11
I20250112 20:51:40.028046 3754 consensus_queue.cc:260] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.028185 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.028386 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.028506 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 8 FOLLOWER]: Advancing to term 9
I20250112 20:51:40.028652 3754 ts_tablet_manager.cc:1428] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20250112 20:51:40.029218 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 11 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.029409 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 10 LEADER]: Stepping down as leader of term 10
I20250112 20:51:40.029500 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.028518 3728 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.029668 3728 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:40.029772 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 10.4, Last appended by leader: 4, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.030079 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 10 FOLLOWER]: Advancing to term 11
I20250112 20:51:40.031112 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.031176 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 11.
I20250112 20:51:40.031566 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.031708 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 9 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.032459 3728 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 FOLLOWER]: Leader election won for term 11
I20250112 20:51:40.032616 3755 tablet_bootstrap.cc:492] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: Bootstrap starting.
I20250112 20:51:40.032619 3728 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.032897 3728 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.033218 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.034190 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.034405 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.034549 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 11 FOLLOWER]: Advancing to term 12
I20250112 20:51:40.035836 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 11 because replica has last-logged OpId of term: 10 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 3.
I20250112 20:51:40.036302 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 10 to 11, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 11 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.036948 3757 tablet_bootstrap.cc:492] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: Bootstrap starting.
I20250112 20:51:40.037532 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.038697 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 12 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.038774 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 12 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.038900 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 11 FOLLOWER]: Advancing to term 12
I20250112 20:51:40.039672 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.039901 3509 raft_consensus.cc:474] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 LEADER]: Not starting forced leader election -- already a leader
I20250112 20:51:40.038697 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.040640 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.040891 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 12 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.041126 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 LEADER]: Stepping down as leader of term 11
I20250112 20:51:40.041250 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.041415 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.041708 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 8 FOLLOWER]: Advancing to term 9
I20250112 20:51:40.041663 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.041870 3757 tablet_bootstrap.cc:654] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.041935 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 12 because replica has last-logged OpId of term: 10 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.042196 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 FOLLOWER]: Advancing to term 12
I20250112 20:51:40.042397 3650 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.042559 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 12 FOLLOWER]: Advancing to term 13
I20250112 20:51:40.034161 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.044741 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 12 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.044828 3510 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 LEADER]: Stepping down as leader of term 8
I20250112 20:51:40.044934 3510 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.045183 3510 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.045454 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 8 FOLLOWER]: Advancing to term 9
I20250112 20:51:40.046017 3650 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.047302 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 9 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20250112 20:51:40.047828 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.047845 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 13 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.048027 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 12 FOLLOWER]: Advancing to term 13
I20250112 20:51:40.049641 3650 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 13 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.050477 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 13.
I20250112 20:51:40.050621 3746 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20250112 20:51:40.051262 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.051594 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 13 FOLLOWER]: Leader election won for term 13
I20250112 20:51:40.052146 3755 tablet_bootstrap.cc:654] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:40.052268 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 13 candidate_status { last_received { term: 10 index: 4 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.052527 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 12 FOLLOWER]: Advancing to term 13
I20250112 20:51:40.053045 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 13 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.053337 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.053471 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 10.4, Last appended by leader: 4, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.054749 3757 tablet_bootstrap.cc:492] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: No bootstrap required, opened a new log
I20250112 20:51:40.053037 3651 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 9.
I20250112 20:51:40.054898 3757 ts_tablet_manager.cc:1397] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.008s
I20250112 20:51:40.054968 3755 tablet_bootstrap.cc:492] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: No bootstrap required, opened a new log
I20250112 20:51:40.055102 3755 ts_tablet_manager.cc:1397] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250112 20:51:40.055191 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: f15597f529254479b5987d2613686dfb
I20250112 20:51:40.055776 3728 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 FOLLOWER]: Leader election won for term 9
I20250112 20:51:40.055939 3728 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.055855 3755 raft_consensus.cc:357] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.056036 3755 raft_consensus.cc:383] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.056123 3755 raft_consensus.cc:738] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.056170 3728 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 4.2, Last appended by leader: 2, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.056844 3757 raft_consensus.cc:357] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.056650 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 11 to 13, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 13 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.057017 3757 raft_consensus.cc:383] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:40.057096 3757 raft_consensus.cc:738] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Initialized, Role: FOLLOWER
I20250112 20:51:40.056349 3755 consensus_queue.cc:260] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.057993 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.057857 3746 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.058101 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 9 FOLLOWER]: Advancing to term 10
I20250112 20:51:40.058200 3746 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:40.057312 3757 consensus_queue.cc:260] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.058858 3722 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.059089 3722 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:40.059551 3757 ts_tablet_manager.cc:1428] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250112 20:51:40.061054 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.062559 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 10 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.062758 3374 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 LEADER]: Stepping down as leader of term 9
I20250112 20:51:40.062384 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 8 to 9, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 9 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.062870 3374 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.063124 3374 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 9.3, Last appended by leader: 3, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.063422 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 9 FOLLOWER]: Advancing to term 10
I20250112 20:51:40.063849 3755 ts_tablet_manager.cc:1428] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250112 20:51:40.066071 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 10 because replica has last-logged OpId of term: 9 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.068077 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 13 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 4.
I20250112 20:51:40.068477 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 10 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.069023 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.068984 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 10 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.069161 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 9 FOLLOWER]: Advancing to term 10
I20250112 20:51:40.069753 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.069916 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.069991 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 13 FOLLOWER]: Advancing to term 14
I20250112 20:51:40.071089 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 10 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.071449 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.071590 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.071743 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20250112 20:51:40.072088 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 14 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.072494 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.072686 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 13 FOLLOWER]: Advancing to term 14
I20250112 20:51:40.072655 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.072826 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 13 LEADER]: Stepping down as leader of term 13
I20250112 20:51:40.072922 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.073172 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 13.5, Last appended by leader: 5, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.073374 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 13 FOLLOWER]: Advancing to term 14
I20250112 20:51:40.074695 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 14.
I20250112 20:51:40.075068 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.075193 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.075325 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 14 because replica has last-logged OpId of term: 13 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 4.
I20250112 20:51:40.075387 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.075389 3728 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 FOLLOWER]: Leader election won for term 14
I20250112 20:51:40.075546 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 10 FOLLOWER]: Advancing to term 11
I20250112 20:51:40.075594 3728 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.075860 3728 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.077733 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.078243 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 11 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.077953 3210 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 13 to 14, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 14 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.078619 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 11 candidate_status { last_received { term: 9 index: 3 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.078837 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 10 FOLLOWER]: Advancing to term 11
I20250112 20:51:40.078780 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 11 candidate_status { last_received { term: 9 index: 3 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.078962 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 10 FOLLOWER]: Advancing to term 11
I20250112 20:51:40.081082 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 11.
I20250112 20:51:40.081216 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 11.
I20250112 20:51:40.081555 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.081614 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.081933 3728 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 FOLLOWER]: Leader election won for term 11
I20250112 20:51:40.082087 3728 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.082345 3728 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 9.3, Last appended by leader: 3, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.082643 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.082803 3510 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.082918 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 14 FOLLOWER]: Advancing to term 15
I20250112 20:51:40.085129 3210 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.085417 3510 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.086063 3510 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 15 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.086416 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 15 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.086602 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 LEADER]: Stepping down as leader of term 14
I20250112 20:51:40.086691 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.086954 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.087186 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 FOLLOWER]: Advancing to term 15
I20250112 20:51:40.087177 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 15 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.087366 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 14 FOLLOWER]: Advancing to term 15
I20250112 20:51:40.087461 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.087682 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.087805 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 11 FOLLOWER]: Advancing to term 12
I20250112 20:51:40.089871 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.090049 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 15 because replica has last-logged OpId of term: 14 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.090277 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 15 because replica has last-logged OpId of term: 13 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.090494 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 12 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.090950 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 12 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.091066 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.091037 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 12 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.091127 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 LEADER]: Stepping down as leader of term 11
I20250112 20:51:40.091228 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 11 FOLLOWER]: Advancing to term 12
I20250112 20:51:40.091228 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.091490 3746 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20250112 20:51:40.091480 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.091742 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 11 FOLLOWER]: Advancing to term 12
I20250112 20:51:40.092990 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.093196 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 12.
I20250112 20:51:40.093538 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.093742 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 12 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 3.
I20250112 20:51:40.093755 3746 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 12 FOLLOWER]: Leader election won for term 12
I20250112 20:51:40.093909 3746 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 12 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.094158 3746 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 8.3, Last appended by leader: 3, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.094997 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.095172 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.095261 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 15 FOLLOWER]: Advancing to term 16
I20250112 20:51:40.096491 3210 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 11 to 12, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 12 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.097648 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.098183 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 16 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.098659 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 16 candidate_status { last_received { term: 13 index: 5 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.098762 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 16 candidate_status { last_received { term: 13 index: 5 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.098935 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 15 FOLLOWER]: Advancing to term 16
I20250112 20:51:40.098939 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 15 FOLLOWER]: Advancing to term 16
I20250112 20:51:40.099139 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.099296 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 12 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.099413 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 12 FOLLOWER]: Advancing to term 13
I20250112 20:51:40.101706 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 13 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.101900 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 16 because replica has last-logged OpId of term: 14 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 5.
I20250112 20:51:40.102377 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 13 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.102551 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 16.
I20250112 20:51:40.102891 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 13 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.103022 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 13 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.103078 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 12 FOLLOWER]: Advancing to term 13
I20250112 20:51:40.103058 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.103224 3510 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 12 LEADER]: Stepping down as leader of term 12
I20250112 20:51:40.103303 3510 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 12 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.103456 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 16 FOLLOWER]: Leader election won for term 16
I20250112 20:51:40.103636 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 16 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.103610 3510 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 12.4, Last appended by leader: 4, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.103896 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.103945 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 12 FOLLOWER]: Advancing to term 13
I20250112 20:51:40.103892 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 13.5, Last appended by leader: 5, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.104916 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.105122 3374 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.105254 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 16 FOLLOWER]: Advancing to term 17
I20250112 20:51:40.106396 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 13 because replica has last-logged OpId of term: 11 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.107138 3374 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.107334 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 13 because replica has last-logged OpId of term: 12 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.107703 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.107738 3374 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 17 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.108004 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20250112 20:51:40.108373 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.108664 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 16 FOLLOWER]: Advancing to term 17
I20250112 20:51:40.108930 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.109110 3374 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.109210 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 13 FOLLOWER]: Advancing to term 14
I20250112 20:51:40.109454 3210 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 14 to 16, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 16 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.111460 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 17.
I20250112 20:51:40.111958 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.112030 3374 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.112475 3728 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 FOLLOWER]: Leader election won for term 17
I20250112 20:51:40.112653 3728 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.112869 3374 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 14 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.112900 3728 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.113158 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.113363 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 16 LEADER]: Stepping down as leader of term 16
I20250112 20:51:40.113380 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.113430 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.113672 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 16.6, Last appended by leader: 6, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.113927 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 16 FOLLOWER]: Advancing to term 17
I20250112 20:51:40.114728 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.114919 3510 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.114921 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.115109 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 17 FOLLOWER]: Advancing to term 18
I20250112 20:51:40.115156 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 13 FOLLOWER]: Advancing to term 14
I20250112 20:51:40.116564 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 17 because replica has last-logged OpId of term: 16 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 5.
I20250112 20:51:40.116295 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 16 to 17, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 17 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.117547 3746 raft_consensus.cc:491] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:40.117600 3510 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.117691 3746 raft_consensus.cc:3054] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.117974 3511 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 14 because replica has last-logged OpId of term: 12 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 4.
I20250112 20:51:40.118891 3510 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 18 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.119307 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 18 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.119446 3374 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 LEADER]: Stepping down as leader of term 17
I20250112 20:51:40.119509 3374 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.119701 3374 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.119899 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 FOLLOWER]: Advancing to term 18
I20250112 20:51:40.119911 3746 raft_consensus.cc:513] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.120389 3746 leader_election.cc:290] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 1 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.120851 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82c4eb8655241348564eafae773b59d" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.121085 3375 raft_consensus.cc:3054] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.122077 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 18 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.122246 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a82c4eb8655241348564eafae773b59d" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.122310 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 14 candidate_status { last_received { term: 11 index: 4 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.122447 3651 raft_consensus.cc:3054] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:40.122591 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 13 FOLLOWER]: Advancing to term 14
I20250112 20:51:40.123836 3375 raft_consensus.cc:2463] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 1.
I20250112 20:51:40.124361 3651 raft_consensus.cc:2463] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 1.
I20250112 20:51:40.124823 3439 leader_election.cc:304] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.125023 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 14.
I20250112 20:51:40.125156 3746 raft_consensus.cc:2798] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:40.125312 3746 raft_consensus.cc:695] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [term 1 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.125720 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: f15597f529254479b5987d2613686dfb
I20250112 20:51:40.125594 3746 consensus_queue.cc:237] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.125785 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.125957 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.125993 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 FOLLOWER]: Leader election won for term 14
I20250112 20:51:40.126056 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 14 FOLLOWER]: Advancing to term 15
I20250112 20:51:40.126127 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.126385 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 11.4, Last appended by leader: 4, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.126896 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 18 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.127210 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 17 FOLLOWER]: Advancing to term 18
I20250112 20:51:40.129848 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 18 because replica has last-logged OpId of term: 16 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.129550 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.130355 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.130676 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 12 to 14, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 14 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.131269 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 15 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.132591 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.132798 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 LEADER]: Stepping down as leader of term 14
I20250112 20:51:40.132921 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.133183 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.133472 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 14 FOLLOWER]: Advancing to term 15
I20250112 20:51:40.134073 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.134963 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.135250 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.135416 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 18 FOLLOWER]: Advancing to term 19
I20250112 20:51:40.135941 3761 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20250112 20:51:40.136173 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 15 because replica has last-logged OpId of term: 14 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 4.
I20250112 20:51:40.137019 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 15 candidate_status { last_received { term: 12 index: 4 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.137212 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 14 FOLLOWER]: Advancing to term 15
I20250112 20:51:40.139073 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 15.
I20250112 20:51:40.139014 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.139575 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.139899 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 19 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.139921 3720 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: TxnStatusTablet state changed. Reason: New leader f15597f529254479b5987d2613686dfb. Latest consensus state: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.140204 3720 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:40.140324 3746 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 15 FOLLOWER]: Leader election won for term 15
I20250112 20:51:40.140466 3746 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 15 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.140771 3746 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 12.4, Last appended by leader: 4, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.140826 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.141018 3650 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.141112 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 15 FOLLOWER]: Advancing to term 16
I20250112 20:51:40.141458 3210 catalog_manager.cc:5526] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.143183 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 19 candidate_status { last_received { term: 16 index: 6 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.143406 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 18 FOLLOWER]: Advancing to term 19
I20250112 20:51:40.143884 3650 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.146050 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 19.
I20250112 20:51:40.146080 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 16 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.146329 3511 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 15 LEADER]: Stepping down as leader of term 15
I20250112 20:51:40.146453 3511 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 15 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.146782 3511 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 15.5, Last appended by leader: 5, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.147220 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 15 FOLLOWER]: Advancing to term 16
I20250112 20:51:40.148883 3650 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 16 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.149703 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 16 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.149919 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 15 FOLLOWER]: Advancing to term 16
I20250112 20:51:40.149718 3210 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 14 to 15, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 15 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.150367 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.150754 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 19 FOLLOWER]: Leader election won for term 19
I20250112 20:51:40.150954 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 19 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.151304 3762 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:40.151219 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 16.6, Last appended by leader: 6, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.151659 3762 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:40.152616 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 16 because replica has last-logged OpId of term: 14 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.153170 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 19 candidate_status { last_received { term: 16 index: 6 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.153436 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 18 FOLLOWER]: Advancing to term 19
I20250112 20:51:40.153800 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.154417 3210 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 17 to 19, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 19 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.154805 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.156436 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 19 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 6.
I20250112 20:51:40.157397 3374 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.157569 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 19 FOLLOWER]: Advancing to term 20
I20250112 20:51:40.161306 3374 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.156708 3511 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 16 because replica has last-logged OpId of term: 15 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.163264 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.163352 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.163496 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 19 FOLLOWER]: Advancing to term 20
I20250112 20:51:40.163550 3650 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 19 LEADER]: Stepping down as leader of term 19
I20250112 20:51:40.163655 3650 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.163820 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.163985 3650 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 19.7, Last appended by leader: 7, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.164327 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 19 FOLLOWER]: Advancing to term 20
I20250112 20:51:40.164443 3731 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20250112 20:51:40.162730 3374 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 20 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.166127 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.166416 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.166543 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 16 FOLLOWER]: Advancing to term 17
I20250112 20:51:40.166648 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 20.
I20250112 20:51:40.167079 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.167397 3728 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 FOLLOWER]: Leader election won for term 20
I20250112 20:51:40.167578 3728 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.168025 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 20 because replica has last-logged OpId of term: 19 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 6.
I20250112 20:51:40.167899 3728 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.170570 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.170591 3210 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 19 to 20, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 20 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.171447 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 17 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.172137 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.172426 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 16 FOLLOWER]: Advancing to term 17
I20250112 20:51:40.174358 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 17 candidate_status { last_received { term: 14 index: 5 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.174587 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 16 FOLLOWER]: Advancing to term 17
I20250112 20:51:40.175516 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.175841 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 17 because replica has last-logged OpId of term: 15 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 5.
I20250112 20:51:40.176601 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.176860 3510 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.177047 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 20 FOLLOWER]: Advancing to term 21
I20250112 20:51:40.177376 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 17.
I20250112 20:51:40.177817 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: f15597f529254479b5987d2613686dfb
I20250112 20:51:40.178139 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 FOLLOWER]: Leader election won for term 17
I20250112 20:51:40.178332 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.178670 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 14.5, Last appended by leader: 5, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.179795 3510 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.180646 3510 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 21 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.181001 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 21 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.181000 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 21 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.181190 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 LEADER]: Stepping down as leader of term 20
I20250112 20:51:40.181310 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.181332 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 20 FOLLOWER]: Advancing to term 21
I20250112 20:51:40.181577 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.181764 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.181874 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 FOLLOWER]: Advancing to term 21
I20250112 20:51:40.181974 3511 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 17 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.182116 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 17 FOLLOWER]: Advancing to term 18
I20250112 20:51:40.184661 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 21 because replica has last-logged OpId of term: 19 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.184819 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 21 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.185010 3511 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 18 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.185706 3511 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 18 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.186239 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 18 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.186076 3210 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 15 to 17, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 17 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.186422 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.186578 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 LEADER]: Stepping down as leader of term 17
I20250112 20:51:40.186717 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.186882 3720 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20250112 20:51:40.187117 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.187489 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 17 FOLLOWER]: Advancing to term 18
I20250112 20:51:40.187706 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 18 candidate_status { last_received { term: 15 index: 5 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.187913 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 17 FOLLOWER]: Advancing to term 18
I20250112 20:51:40.189263 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.190093 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.190320 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.190438 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 21 FOLLOWER]: Advancing to term 22
I20250112 20:51:40.192338 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 18.
I20250112 20:51:40.192859 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.193222 3720 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 18 FOLLOWER]: Leader election won for term 18
I20250112 20:51:40.193212 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.193382 3720 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 18 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.193715 3720 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 15.5, Last appended by leader: 5, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.194296 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.194509 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 21 FOLLOWER]: Advancing to term 22
I20250112 20:51:40.195138 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 22 candidate_status { last_received { term: 19 index: 7 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.195319 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 21 FOLLOWER]: Advancing to term 22
I20250112 20:51:40.196007 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 22 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.197147 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 22 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 7.
I20250112 20:51:40.197412 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.197144 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 18 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 5.
I20250112 20:51:40.197691 3650 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.197908 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 18 FOLLOWER]: Advancing to term 19
I20250112 20:51:40.198839 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 22.
I20250112 20:51:40.198678 3210 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 17 to 18, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 18 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.199227 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.199676 3731 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 22 FOLLOWER]: Leader election won for term 22
I20250112 20:51:40.199836 3731 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 22 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.200148 3731 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 19.7, Last appended by leader: 7, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.202765 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 20 to 22, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 22 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.203833 3650 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.205052 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 19 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.205058 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 19 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.205261 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 18 FOLLOWER]: Advancing to term 19
I20250112 20:51:40.205399 3510 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 18 LEADER]: Stepping down as leader of term 18
I20250112 20:51:40.205551 3510 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 18 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.205724 3650 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 19 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.206087 3510 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 18.6, Last appended by leader: 6, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.206601 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 18 FOLLOWER]: Advancing to term 19
I20250112 20:51:40.208755 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 19 because replica has last-logged OpId of term: 17 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.209602 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 19 because replica has last-logged OpId of term: 18 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.209949 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.210024 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.210341 3731 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20250112 20:51:40.210800 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.210983 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.211081 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 22 FOLLOWER]: Advancing to term 23
I20250112 20:51:40.212922 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.213464 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 23 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.214057 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.214084 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.214291 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 22 FOLLOWER]: Advancing to term 23
I20250112 20:51:40.214617 3650 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 22 LEADER]: Stepping down as leader of term 22
I20250112 20:51:40.214668 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.214776 3650 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.214859 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.214987 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 19 FOLLOWER]: Advancing to term 20
I20250112 20:51:40.215236 3650 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 22.8, Last appended by leader: 8, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.215616 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 22 FOLLOWER]: Advancing to term 23
I20250112 20:51:40.217265 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 23.
I20250112 20:51:40.217792 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.217796 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.218174 3721 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 FOLLOWER]: Leader election won for term 23
I20250112 20:51:40.218317 3721 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.218633 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 20 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.218940 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 23 because replica has last-logged OpId of term: 22 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 7.
I20250112 20:51:40.219404 3721 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.219516 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.219558 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 20 candidate_status { last_received { term: 17 index: 6 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.219733 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 19 FOLLOWER]: Advancing to term 20
I20250112 20:51:40.219748 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 19 FOLLOWER]: Advancing to term 20
I20250112 20:51:40.222579 3511 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 20 because replica has last-logged OpId of term: 18 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 6.
I20250112 20:51:40.222577 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 20.
I20250112 20:51:40.222544 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 22 to 23, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 23 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.223172 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.223323 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:40.224130 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.224367 3509 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.224493 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 23 FOLLOWER]: Advancing to term 24
I20250112 20:51:40.226872 3509 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.227907 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 FOLLOWER]: Leader election won for term 20
I20250112 20:51:40.227907 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 24 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.227980 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 24 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.228114 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.228156 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 LEADER]: Stepping down as leader of term 23
I20250112 20:51:40.228202 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 23 FOLLOWER]: Advancing to term 24
I20250112 20:51:40.228226 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.228392 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 17.6, Last appended by leader: 6, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.228459 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.228717 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 FOLLOWER]: Advancing to term 24
I20250112 20:51:40.229815 3509 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 24 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.231122 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.231400 3509 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.231515 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 24 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.231657 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 20 FOLLOWER]: Advancing to term 21
I20250112 20:51:40.231999 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 18 to 20, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 20 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.234238 3509 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.234880 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 24 because replica has last-logged OpId of term: 22 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.235342 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 21 candidate_status { last_received { term: 18 index: 6 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.235473 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.235674 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 LEADER]: Stepping down as leader of term 20
I20250112 20:51:40.235807 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.235875 3761 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20250112 20:51:40.236191 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.236475 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 20 FOLLOWER]: Advancing to term 21
I20250112 20:51:40.236445 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 21 candidate_status { last_received { term: 18 index: 6 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.236657 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 20 FOLLOWER]: Advancing to term 21
I20250112 20:51:40.239280 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 21.
I20250112 20:51:40.239368 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 21 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 6.
I20250112 20:51:40.239500 3509 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 21 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.239792 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.240087 3761 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 21 FOLLOWER]: Leader election won for term 21
I20250112 20:51:40.240255 3761 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 21 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.240600 3761 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 18.6, Last appended by leader: 6, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.243620 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 20 to 21, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 21 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.247243 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.248029 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.248231 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.248345 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 24 FOLLOWER]: Advancing to term 25
I20250112 20:51:40.250422 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.251060 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 25 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.251482 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 25 candidate_status { last_received { term: 22 index: 8 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.251598 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 25 candidate_status { last_received { term: 22 index: 8 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.251717 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 24 FOLLOWER]: Advancing to term 25
I20250112 20:51:40.251827 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 24 FOLLOWER]: Advancing to term 25
I20250112 20:51:40.252493 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.252775 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 21 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.252954 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 21 FOLLOWER]: Advancing to term 22
I20250112 20:51:40.254683 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 25 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 8.
I20250112 20:51:40.254739 3509 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 25.
I20250112 20:51:40.255237 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.255504 3731 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 25 FOLLOWER]: Leader election won for term 25
I20250112 20:51:40.255666 3731 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 25 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.255865 3731 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 22.8, Last appended by leader: 8, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.255995 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 22 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.256899 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 22 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.257614 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 22 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.257635 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 22 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.257833 3509 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 21 LEADER]: Stepping down as leader of term 21
I20250112 20:51:40.257942 3509 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 21 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.258225 3509 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 21.7, Last appended by leader: 7, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.258569 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 21 FOLLOWER]: Advancing to term 22
I20250112 20:51:40.258437 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 23 to 25, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 25 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.259389 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 21 FOLLOWER]: Advancing to term 22
I20250112 20:51:40.261739 3509 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 22 because replica has last-logged OpId of term: 21 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.262741 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 22 because replica has last-logged OpId of term: 20 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.263165 3578 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.263505 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20250112 20:51:40.264605 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.265492 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.265691 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.265800 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 25 FOLLOWER]: Advancing to term 26
I20250112 20:51:40.267793 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.268340 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 26 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.268945 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.269078 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.269204 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 25 FOLLOWER]: Advancing to term 26
I20250112 20:51:40.269320 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 25 LEADER]: Stepping down as leader of term 25
I20250112 20:51:40.269418 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.269503 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.269719 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 22 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.269722 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 25.9, Last appended by leader: 9, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.269863 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 22 FOLLOWER]: Advancing to term 23
I20250112 20:51:40.270025 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 25 FOLLOWER]: Advancing to term 26
I20250112 20:51:40.272495 3509 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 26.
I20250112 20:51:40.272459 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.272864 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 26 because replica has last-logged OpId of term: 25 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 8.
I20250112 20:51:40.273195 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 23 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.273160 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.273506 3728 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 FOLLOWER]: Leader election won for term 26
I20250112 20:51:40.273679 3728 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.273905 3728 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.274189 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.274375 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 22 FOLLOWER]: Advancing to term 23
I20250112 20:51:40.276808 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.277645 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 23 candidate_status { last_received { term: 20 index: 7 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.277913 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 22 FOLLOWER]: Advancing to term 23
I20250112 20:51:40.277920 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.278144 3509 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.278260 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 26 FOLLOWER]: Advancing to term 27
I20250112 20:51:40.278282 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 25 to 26, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 26 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.280997 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 23.
I20250112 20:51:40.281037 3509 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.281682 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:40.282063 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 FOLLOWER]: Leader election won for term 23
I20250112 20:51:40.282162 3511 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 23 because replica has last-logged OpId of term: 21 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 7.
I20250112 20:51:40.282207 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.282492 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 20.7, Last appended by leader: 7, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.282645 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.282817 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 LEADER]: Stepping down as leader of term 26
I20250112 20:51:40.282902 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.283130 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.283373 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 FOLLOWER]: Advancing to term 27
I20250112 20:51:40.285239 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.285069 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 21 to 23, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 23 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.285518 3509 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 27 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.285697 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 26 FOLLOWER]: Advancing to term 27
I20250112 20:51:40.286511 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 27 because replica has last-logged OpId of term: 26 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.287114 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.287312 3509 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.287444 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 23 FOLLOWER]: Advancing to term 24
I20250112 20:51:40.289273 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 27 because replica has last-logged OpId of term: 25 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.289794 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.290102 3761 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20250112 20:51:40.290232 3509 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.290820 3509 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 24 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.291272 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 24 candidate_status { last_received { term: 21 index: 7 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.291493 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 23 FOLLOWER]: Advancing to term 24
I20250112 20:51:40.291429 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 24 candidate_status { last_received { term: 21 index: 7 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.291663 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 LEADER]: Stepping down as leader of term 23
I20250112 20:51:40.291754 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.292027 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.292275 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 23 FOLLOWER]: Advancing to term 24
I20250112 20:51:40.294317 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 24.
I20250112 20:51:40.294688 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 24 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 7.
I20250112 20:51:40.295266 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.295334 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.295814 3761 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 24 FOLLOWER]: Leader election won for term 24
I20250112 20:51:40.295970 3761 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 24 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.296244 3761 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 21.7, Last appended by leader: 7, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.296720 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.296976 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 27 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.297158 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 27 FOLLOWER]: Advancing to term 28
I20250112 20:51:40.299803 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 28 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.300915 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 28 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.301254 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.301499 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 27 FOLLOWER]: Advancing to term 28
I20250112 20:51:40.304396 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 28 because replica has last-logged OpId of term: 26 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 9.
I20250112 20:51:40.299922 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 23 to 24, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 24 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.307597 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 28 candidate_status { last_received { term: 25 index: 9 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.307855 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 27 FOLLOWER]: Advancing to term 28
I20250112 20:51:40.310880 3509 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 28.
I20250112 20:51:40.310832 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.311162 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 24 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.311332 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 24 FOLLOWER]: Advancing to term 25
I20250112 20:51:40.312283 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.312932 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 28 FOLLOWER]: Leader election won for term 28
I20250112 20:51:40.313131 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 28 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.313405 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 25.9, Last appended by leader: 9, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.314708 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 25 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.315820 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 25 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.316830 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 25 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.317070 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 24 FOLLOWER]: Advancing to term 25
I20250112 20:51:40.317093 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 26 to 28, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 28 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.317555 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.318338 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.318548 3374 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.318675 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 FOLLOWER]: Advancing to term 29
I20250112 20:51:40.321444 3374 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.322547 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 29 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.322809 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 28 FOLLOWER]: Advancing to term 29
I20250112 20:51:40.325740 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 29.
I20250112 20:51:40.332929 3374 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 29 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.332978 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.333386 3728 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 FOLLOWER]: Leader election won for term 29
I20250112 20:51:40.333343 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 25 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.333547 3728 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.333566 3509 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 24 LEADER]: Stepping down as leader of term 24
I20250112 20:51:40.333685 3509 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 24 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.333884 3728 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.333988 3509 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 24.8, Last appended by leader: 8, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.334271 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 24 FOLLOWER]: Advancing to term 25
I20250112 20:51:40.336191 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 29 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.336406 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 28 LEADER]: Stepping down as leader of term 28
I20250112 20:51:40.336508 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.336787 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.337131 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 28 FOLLOWER]: Advancing to term 29
I20250112 20:51:40.337215 3509 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 25 because replica has last-logged OpId of term: 24 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.339119 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 28 to 29, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 29 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.339818 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 25 because replica has last-logged OpId of term: 23 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.341518 3650 raft_consensus.cc:1270] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96 [term 1 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:40.341938 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 29 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 9.
I20250112 20:51:40.342075 3578 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.342245 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.342473 3374 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.342600 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 25 FOLLOWER]: Advancing to term 26
I20250112 20:51:40.343672 3761 consensus_queue.cc:1035] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.345005 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20250112 20:51:40.345332 3374 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.346000 3374 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 26 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.347498 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.347719 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 25 FOLLOWER]: Advancing to term 26
I20250112 20:51:40.349977 3375 raft_consensus.cc:1270] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 1 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:40.350621 3761 consensus_queue.cc:1035] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.352663 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.353560 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.353760 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.353909 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 29 FOLLOWER]: Advancing to term 30
I20250112 20:51:40.355298 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 26.
I20250112 20:51:40.355772 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:40.356048 3728 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 FOLLOWER]: Leader election won for term 26
I20250112 20:51:40.356904 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.358753 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 30 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.358912 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.359881 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.359175 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 29 FOLLOWER]: Advancing to term 30
I20250112 20:51:40.360173 3511 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 25 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.360330 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 25 FOLLOWER]: Advancing to term 26
I20250112 20:51:40.361563 3722 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: TxnStatusTablet state changed. Reason: New leader f15597f529254479b5987d2613686dfb. Latest consensus state: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.361837 3722 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:40.362545 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.362751 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 LEADER]: Stepping down as leader of term 29
I20250112 20:51:40.362872 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.363842 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 29.10, Last appended by leader: 10, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.363911 3511 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 26 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.364176 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 FOLLOWER]: Advancing to term 30
I20250112 20:51:40.365244 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 26 candidate_status { last_received { term: 24 index: 8 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.366658 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 26 candidate_status { last_received { term: 24 index: 8 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.366976 3652 raft_consensus.cc:2388] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb in current term 26: Already voted for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in this term.
I20250112 20:51:40.367391 3511 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 26 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.367796 3765 mvcc.cc:204] Tried to move back new op lower bound from 7113585050988085248 to 7113585050117599232. Current Snapshot: MvccSnapshot[applied={T|T < 7113585050988085248}]
I20250112 20:51:40.370229 3728 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.370755 3728 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 23.8, Last appended by leader: 8, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.372180 3373 raft_consensus.cc:2388] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 LEADER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb in current term 26: Already voted for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in this term.
I20250112 20:51:40.372918 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.373442 3720 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20250112 20:51:40.375056 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 30 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.375061 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.376096 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.376322 3651 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 30 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.376454 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 30 FOLLOWER]: Advancing to term 31
I20250112 20:51:40.378350 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 26 candidate_status { last_received { term: 23 index: 8 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.378638 3509 raft_consensus.cc:2388] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in current term 26: Already voted for candidate f15597f529254479b5987d2613686dfb in this term.
I20250112 20:51:40.379540 3651 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 31 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.380777 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.380972 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 30 FOLLOWER]: Advancing to term 31
I20250112 20:51:40.381232 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.381839 3651 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 31 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.382752 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.382933 3650 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.383037 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 26 FOLLOWER]: Advancing to term 27
I20250112 20:51:40.384975 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 30 because replica has last-logged OpId of term: 29 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.385491 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.385998 3650 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.386955 3373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.387151 3373 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 LEADER]: Stepping down as leader of term 26
I20250112 20:51:40.387234 3373 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.387969 3373 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.388273 3373 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 26 FOLLOWER]: Advancing to term 27
I20250112 20:51:40.393795 3373 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 27 because replica has last-logged OpId of term: 26 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.426653 3722 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.426962 3722 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P 710959530a634cc991544c14efad9e96: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:40.429128 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 27 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.429411 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 26 FOLLOWER]: Advancing to term 27
I20250112 20:51:40.432351 3511 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 27 because replica has last-logged OpId of term: 24 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.432855 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.433558 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20250112 20:51:40.438271 3509 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 31.
I20250112 20:51:40.438570 3720 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20250112 20:51:40.439667 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.440014 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 31 FOLLOWER]: Leader election won for term 31
I20250112 20:51:40.440171 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 31 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.440503 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.444113 3721 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: TxnStatusTablet state changed. Reason: New leader f15597f529254479b5987d2613686dfb. Latest consensus state: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.444407 3721 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:40.444972 3761 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.445241 3761 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:40.445582 3650 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 27 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.445312 3210 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 29 to 31, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 31 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.452571 3746 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.452891 3746 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P f15597f529254479b5987d2613686dfb: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:40.457281 3768 mvcc.cc:204] Tried to move back new op lower bound from 7113585050988085248 to 7113585050117599232. Current Snapshot: MvccSnapshot[applied={T|T < 7113585050988085248}]
I20250112 20:51:40.458954 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.459774 3373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.460022 3373 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 30 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.460139 3373 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 30 FOLLOWER]: Advancing to term 31
I20250112 20:51:40.463773 3373 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.465103 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 31 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.465469 3509 raft_consensus.cc:2388] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in current term 31: Already voted for candidate 710959530a634cc991544c14efad9e96 in this term.
I20250112 20:51:40.466912 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 31 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.467276 3652 raft_consensus.cc:2388] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 31 LEADER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in current term 31: Already voted for candidate 710959530a634cc991544c14efad9e96 in this term.
I20250112 20:51:40.467949 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.468379 3728 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20250112 20:51:40.468611 3373 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 31 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.470077 3373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.470289 3373 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 27 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.470441 3373 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 27 FOLLOWER]: Advancing to term 28
I20250112 20:51:40.511636 3373 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.513295 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 28 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.513626 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 27 FOLLOWER]: Advancing to term 28
I20250112 20:51:40.517622 3509 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 28.
I20250112 20:51:40.518146 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.518585 3728 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 FOLLOWER]: Leader election won for term 28
I20250112 20:51:40.518769 3728 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.519132 3728 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 26.9, Last appended by leader: 9, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.521834 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 28 candidate_status { last_received { term: 26 index: 9 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.522104 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 27 FOLLOWER]: Advancing to term 28
I20250112 20:51:40.525913 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 28.
I20250112 20:51:40.527068 3373 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 28 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.528843 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.529708 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.529937 3509 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.530076 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Advancing to term 32
I20250112 20:51:40.530440 3721 tablet_replica.cc:440] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.530651 3721 tablet_replica.cc:443] T a82c4eb8655241348564eafae773b59d P d3f9c3f8fe0d4459b54d92f75a2c84b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:40.534582 3509 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.534734 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 24 to 28, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 28 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.535326 3509 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 32 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.536051 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 32 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.536293 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 FOLLOWER]: Advancing to term 32
I20250112 20:51:40.539420 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 32 because replica has last-logged OpId of term: 29 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.543452 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.550814 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 32 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.551070 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 31 LEADER]: Stepping down as leader of term 31
I20250112 20:51:40.551156 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.551424 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 31.11, Last appended by leader: 11, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.551714 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 31 FOLLOWER]: Advancing to term 32
I20250112 20:51:40.553124 3509 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 28 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.553332 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 28 FOLLOWER]: Advancing to term 29
I20250112 20:51:40.554651 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 32 because replica has last-logged OpId of term: 31 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20250112 20:51:40.554711 3784 tablet_replica.cc:1307] Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER: cancelling pending write operations
W20250112 20:51:40.555510 3783 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:40.555878 3784 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.559463 3509 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 29 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.559913 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:40.560758 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.561071 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 29 candidate_status { last_received { term: 24 index: 8 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.561271 3374 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 LEADER]: Stepping down as leader of term 28
I20250112 20:51:40.561379 3374 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.561652 3374 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.561740 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.561970 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 28 FOLLOWER]: Advancing to term 29
I20250112 20:51:40.562008 3761 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250112 20:51:40.565243 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 29 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 8.
I20250112 20:51:40.572131 3509 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 29 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
W20250112 20:51:40.575250 3790 tablet_replica.cc:1307] Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER: cancelling pending write operations
W20250112 20:51:40.575506 3790 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.577425 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 29 candidate_status { last_received { term: 24 index: 8 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.577661 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 28 FOLLOWER]: Advancing to term 29
I20250112 20:51:40.578797 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.580125 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.580312 3650 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 32 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.580449 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 32 FOLLOWER]: Advancing to term 33
I20250112 20:51:40.580473 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 29.
I20250112 20:51:40.580952 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.581427 3761 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 29 FOLLOWER]: Leader election won for term 29
I20250112 20:51:40.581684 3761 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 29 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.582554 3761 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 24.8, Last appended by leader: 8, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.583787 3650 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 33 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.584900 3650 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 33 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.585261 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 33 candidate_status { last_received { term: 31 index: 11 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.585454 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 32 FOLLOWER]: Advancing to term 33
I20250112 20:51:40.589226 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.589466 3650 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 29 FOLLOWER]: Starting forced leader election (received explicit request)
W20250112 20:51:40.589442 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.589583 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 29 FOLLOWER]: Advancing to term 30
I20250112 20:51:40.589969 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 28 to 29, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 29 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.592602 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 33 candidate_status { last_received { term: 31 index: 11 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.592783 3650 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 30 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.593266 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.593869 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.594069 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 29 FOLLOWER]: Advancing to term 30
I20250112 20:51:40.595116 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Advancing to term 33
W20250112 20:51:40.595722 3788 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.597441 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 30 because replica has last-logged OpId of term: 28 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.597949 3374 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 33.
W20250112 20:51:40.598126 3787 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.598347 3578 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:40.598723 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 33 FOLLOWER]: Leader election won for term 33
I20250112 20:51:40.598896 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 33 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.599236 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 31.11, Last appended by leader: 11, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.599978 3798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.602931 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.603338 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 30 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.603619 3511 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 29 LEADER]: Stepping down as leader of term 29
I20250112 20:51:40.603718 3511 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 29 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.598357 3650 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 30 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.604043 3511 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 29.9, Last appended by leader: 9, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.604362 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 29 FOLLOWER]: Advancing to term 30
W20250112 20:51:40.607411 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.608142 3509 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 33.
W20250112 20:51:40.609150 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.609298 3511 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 30 because replica has last-logged OpId of term: 29 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.610191 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
W20250112 20:51:40.610373 3790 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.610539 3731 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20250112 20:51:40.611202 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.612679 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.612905 3374 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.613015 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 FOLLOWER]: Advancing to term 34
I20250112 20:51:40.615984 3374 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.617158 3374 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 34 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.617674 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 34 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.617919 3650 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 33 LEADER]: Stepping down as leader of term 33
I20250112 20:51:40.618017 3650 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.618299 3650 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 33.12, Last appended by leader: 12, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.618634 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 33 FOLLOWER]: Advancing to term 34
I20250112 20:51:40.635299 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 34 because replica has last-logged OpId of term: 33 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 10.
I20250112 20:51:40.637184 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 34 candidate_status { last_received { term: 29 index: 10 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.637426 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: Advancing to term 34
I20250112 20:51:40.639577 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.639812 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 30 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.639971 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 30 FOLLOWER]: Advancing to term 31
I20250112 20:51:40.642916 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.644081 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.644327 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 30 FOLLOWER]: Advancing to term 31
I20250112 20:51:40.644678 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 34.
I20250112 20:51:40.647326 3509 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 31 because replica has last-logged OpId of term: 29 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 10.
I20250112 20:51:40.647854 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96
I20250112 20:51:40.647935 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 31 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.648813 3721 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 FOLLOWER]: Leader election won for term 34
I20250112 20:51:40.648964 3721 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.649258 3721 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 29.10, Last appended by leader: 10, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.649586 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 31 candidate_status { last_received { term: 28 index: 10 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.649796 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 30 FOLLOWER]: Advancing to term 31
I20250112 20:51:40.652102 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 33 to 34, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 34 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.656605 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.657532 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.657757 3509 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.657893 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 34 FOLLOWER]: Advancing to term 35
W20250112 20:51:40.659303 3804 tablet_replica.cc:1307] Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER: cancelling pending write operations
W20250112 20:51:40.659600 3808 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.659874 3802 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.660488 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.661089 3509 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.661871 3803 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.662249 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 35 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.662531 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 31.
I20250112 20:51:40.662569 3375 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 LEADER]: Stepping down as leader of term 34
W20250112 20:51:40.662688 3806 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.663004 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: f15597f529254479b5987d2613686dfb
I20250112 20:51:40.663288 3509 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 35 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
W20250112 20:51:40.663422 3807 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.663527 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 FOLLOWER]: Leader election won for term 31
I20250112 20:51:40.663664 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.663955 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 28.10, Last appended by leader: 10, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.664366 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.664582 3509 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.664687 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 31 FOLLOWER]: Advancing to term 32
I20250112 20:51:40.665665 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 35 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.665877 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 34 FOLLOWER]: Advancing to term 35
I20250112 20:51:40.666972 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 29 to 31, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 31 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:40.668601 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.662804 3375 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
W20250112 20:51:40.670593 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:40.671277 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.671190 3375 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 34.11, Last appended by leader: 11, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.671566 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 FOLLOWER]: Advancing to term 35
I20250112 20:51:40.674367 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 35 because replica has last-logged OpId of term: 34 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20250112 20:51:40.674547 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.678499 3652 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 31 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 31 index: 12. (index mismatch)
I20250112 20:51:40.679162 3728 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:40.685880 3799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.686780 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.687944 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.692384 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.699187 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:40.700361 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.701454 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 35 because replica has last-logged OpId of term: 33 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250112 20:51:40.701982 3509 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.702443 3511 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 31. Current term is 32. Ops: []
I20250112 20:51:40.702740 3509 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 32 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.703279 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 32 candidate_status { last_received { term: 29 index: 9 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.703481 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 LEADER]: Stepping down as leader of term 31
I20250112 20:51:40.703591 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.703971 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.704237 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 31 FOLLOWER]: Advancing to term 32
W20250112 20:51:40.704440 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.704535 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.705394 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.705631 3650 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.705814 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 35 FOLLOWER]: Advancing to term 36
I20250112 20:51:40.707160 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 9.
I20250112 20:51:40.708863 3650 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.709785 3650 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 36 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.711083 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 36 candidate_status { last_received { term: 33 index: 12 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.711337 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 35 FOLLOWER]: Advancing to term 36
W20250112 20:51:40.705559 3769 consensus_queue.cc:1159] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f15597f529254479b5987d2613686dfb" responder_term: 32 status { last_received { term: 29 index: 9 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 31. Current term is 32. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250112 20:51:40.712827 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
W20250112 20:51:40.713035 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.713115 3650 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 31 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.713340 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 31 FOLLOWER]: Advancing to term 32
I20250112 20:51:40.714607 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 36.
I20250112 20:51:40.714732 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 32 candidate_status { last_received { term: 29 index: 9 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.715081 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.715634 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 36 candidate_status { last_received { term: 33 index: 12 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.715853 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 FOLLOWER]: Advancing to term 36
I20250112 20:51:40.717115 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.717458 3746 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20250112 20:51:40.717625 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.717885 3746 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250112 20:51:40.718221 3650 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 32 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.719822 3811 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.720127 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.719337 3814 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 36 FOLLOWER]: Leader election won for term 36
W20250112 20:51:40.720364 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.720394 3814 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 36 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.720381 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.720669 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 15.
I20250112 20:51:40.720772 3814 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 33.12, Last appended by leader: 12, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.721011 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.721271 3511 raft_consensus.cc:2388] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 in current term 32: Already voted for candidate f15597f529254479b5987d2613686dfb in this term.
I20250112 20:51:40.721709 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.721931 3650 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 32 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.722018 3731 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20250112 20:51:40.722926 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 36 because replica has last-logged OpId of term: 34 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 12.
I20250112 20:51:40.724030 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 34 to 36, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 36 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:40.725173 3375 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 34 index: 11. Preceding OpId from leader: term: 36 index: 14. (index mismatch)
I20250112 20:51:40.725216 3511 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 36 index: 14. (index mismatch)
I20250112 20:51:40.725744 3731 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.725976 3731 consensus_queue.cc:1225] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 34.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250112 20:51:40.726130 3722 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.727350 3375 pending_rounds.cc:77] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 0
I20250112 20:51:40.727496 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250112 20:51:40.727622 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.727730 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 5.2
I20250112 20:51:40.727828 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.727918 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 8.3
I20250112 20:51:40.728001 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.728080 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 11.4
I20250112 20:51:40.728157 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.728230 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 14.5
I20250112 20:51:40.728302 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.728372 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 17.6
I20250112 20:51:40.728442 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.728514 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 20.7
I20250112 20:51:40.728607 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.728681 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 23.8
I20250112 20:51:40.728749 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.728837 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 26.9
I20250112 20:51:40.728905 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.728968 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 29.10
I20250112 20:51:40.729028 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.729105 3375 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 34.11
I20250112 20:51:40.729158 3375 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.735392 3766 mvcc.cc:204] Tried to move back new op lower bound from 7113585052563664896 to 7113585048737214464. Current Snapshot: MvccSnapshot[applied={T|T < 7113585052563664896}]
I20250112 20:51:40.735698 3766 mvcc.cc:204] Tried to move back new op lower bound from 7113585052563664896 to 7113585051412848640. Current Snapshot: MvccSnapshot[applied={T|T < 7113585052563664896}]
I20250112 20:51:40.746677 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.747453 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.747654 3374 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.747802 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: Advancing to term 37
I20250112 20:51:40.750509 3374 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.751662 3375 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 36. Current term is 37. Ops: [36.18-36.18]
I20250112 20:51:40.752830 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 37 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.753036 3650 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 36 LEADER]: Stepping down as leader of term 36
I20250112 20:51:40.753129 3650 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.753371 3650 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.20, Last appended by leader: 20, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.753628 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 36 FOLLOWER]: Advancing to term 37
I20250112 20:51:40.754752 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 37 candidate_status { last_received { term: 36 index: 17 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.756330 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 37 because replica has last-logged OpId of term: 36 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 17.
I20250112 20:51:40.757040 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.757160 3374 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 37 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.757403 3816 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20250112 20:51:40.760066 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.760386 3374 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 32 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.760522 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 32 FOLLOWER]: Advancing to term 33
W20250112 20:51:40.763458 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.765242 3374 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.765933 3374 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 33 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.766501 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.766753 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 32 FOLLOWER]: Advancing to term 33
I20250112 20:51:40.768977 3509 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 33.
I20250112 20:51:40.768972 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.769351 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 32 FOLLOWER]: Advancing to term 33
I20250112 20:51:40.769459 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.769726 3816 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 FOLLOWER]: Leader election won for term 33
I20250112 20:51:40.769886 3816 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.770175 3816 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.772871 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.772845 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.774688 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 33.
W20250112 20:51:40.781078 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.791033 3650 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 33 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 31 index: 15. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20250112 20:51:40.791059 3511 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 29 index: 9. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20250112 20:51:40.791503 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.791718 3816 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 13, Time since last communication: 0.000s
I20250112 20:51:40.792143 3812 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:40.792333 3812 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer f15597f529254479b5987d2613686dfb log is divergent from this leader: its last log entry 29.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250112 20:51:40.793051 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.793262 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.793427 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Advancing to term 37
I20250112 20:51:40.793956 3509 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting all ops after (but not including) 0
I20250112 20:51:40.794140 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250112 20:51:40.794292 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.794415 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 5.2
I20250112 20:51:40.794498 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.794584 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 8.3
I20250112 20:51:40.794659 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.794755 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 12.4
I20250112 20:51:40.794852 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.794929 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 15.5
I20250112 20:51:40.795001 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.795089 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 18.6
I20250112 20:51:40.795183 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.795264 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 21.7
I20250112 20:51:40.795354 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.795444 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 24.8
I20250112 20:51:40.795521 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.795603 3509 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb: Aborting uncommitted NO_OP operation due to leader change: 29.9
I20250112 20:51:40.795686 3509 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.796739 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 37 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.798060 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 37 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.799259 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 37 candidate_status { last_received { term: 36 index: 16 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.799551 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 37 because replica has last-logged OpId of term: 36 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 16.
I20250112 20:51:40.804629 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.809284 3817 mvcc.cc:204] Tried to move back new op lower bound from 7113585052374138880 to 7113585051729625088. Current Snapshot: MvccSnapshot[applied={T|T < 7113585052374138880}]
I20250112 20:51:40.809357 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 37 candidate_status { last_received { term: 36 index: 16 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.809901 3374 raft_consensus.cc:2388] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb in current term 37: Already voted for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in this term.
I20250112 20:51:40.810407 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.811189 3819 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20250112 20:51:40.834856 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.835574 3511 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.835742 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 33 FOLLOWER]: Advancing to term 34
I20250112 20:51:40.838862 3511 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.839787 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.840632 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.840693 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.840854 3374 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 LEADER]: Stepping down as leader of term 33
I20250112 20:51:40.840947 3374 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.841063 3511 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 34 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.841230 3374 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.28, Last appended by leader: 28, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.841477 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 33 FOLLOWER]: Advancing to term 34
I20250112 20:51:40.843943 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 34 because replica has last-logged OpId of term: 33 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
I20250112 20:51:40.844470 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.845057 3746 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20250112 20:51:40.847854 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.856678 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
W20250112 20:51:40.857602 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.857553 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.857806 3650 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.857986 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Advancing to term 38
I20250112 20:51:40.860869 3650 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 38 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.861732 3650 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 38 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.862308 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.862562 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Advancing to term 38
I20250112 20:51:40.865070 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 38 candidate_status { last_received { term: 36 index: 20 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.865283 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 37 FOLLOWER]: Advancing to term 38
I20250112 20:51:40.865500 3374 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 38.
I20250112 20:51:40.866075 3578 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:40.866475 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 38 FOLLOWER]: Leader election won for term 38
I20250112 20:51:40.866668 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 38 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.866952 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.20, Last appended by leader: 20, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.868054 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.868255 3650 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 33 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.868363 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 33 FOLLOWER]: Advancing to term 34
I20250112 20:51:40.869098 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 38.
I20250112 20:51:40.871222 3650 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 34 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.872283 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.872587 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 34 because replica has last-logged OpId of term: 33 index: 28, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
W20250112 20:51:40.872864 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.873245 3650 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 34 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.874054 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.874306 3510 raft_consensus.cc:2388] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 in current term 34: Already voted for candidate f15597f529254479b5987d2613686dfb in this term.
I20250112 20:51:40.874655 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:40.875087 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20250112 20:51:40.875252 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 36 to 38. New cstate: current_term: 38 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:40.884130 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.897329 3510 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 38 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 36 index: 16. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20250112 20:51:40.897374 3374 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 36 index: 17. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20250112 20:51:40.897871 3722 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 14, Time since last communication: 0.000s
I20250112 20:51:40.898531 3722 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 15, Time since last communication: 0.000s
I20250112 20:51:40.900825 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:40.901731 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.903590 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.903754 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 FOLLOWER]: Advancing to term 39
I20250112 20:51:40.906487 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.907894 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.908015 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 39 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.908169 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 38 FOLLOWER]: Advancing to term 39
I20250112 20:51:40.910979 3511 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 39.
I20250112 20:51:40.911423 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.911805 3812 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 FOLLOWER]: Leader election won for term 39
I20250112 20:51:40.911942 3812 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.912140 3511 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 38. Current term is 39. Ops: []
I20250112 20:51:40.912259 3812 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 38.22, Last appended by leader: 22, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.912636 3731 consensus_queue.cc:1046] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 17, Time since last communication: 0.000s
I20250112 20:51:40.914903 3374 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 LEADER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 38. Current term is 39. Ops: []
I20250112 20:51:40.915573 3731 consensus_queue.cc:1046] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: INVALID_TERM, Last received: 38.22, Next index: 23, Last known committed idx: 17, Time since last communication: 0.000s
I20250112 20:51:40.915843 3731 consensus_queue.cc:1225] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 39.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20250112 20:51:40.918778 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 38 to 39, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 39 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.919286 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 39 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.920843 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:40.921087 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.921224 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 34 FOLLOWER]: Advancing to term 35
W20250112 20:51:40.923130 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.923919 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.924757 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 35 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.925287 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 35 candidate_status { last_received { term: 33 index: 28 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.925458 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 34 FOLLOWER]: Advancing to term 35
I20250112 20:51:40.925420 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 35 candidate_status { last_received { term: 33 index: 28 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.925611 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 34 FOLLOWER]: Advancing to term 35
I20250112 20:51:40.928241 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 35.
I20250112 20:51:40.928984 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:40.929242 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:40.929256 3812 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 FOLLOWER]: Leader election won for term 35
I20250112 20:51:40.929442 3812 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.929536 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 35.
I20250112 20:51:40.929723 3812 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.28, Last appended by leader: 28, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.932921 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 33 to 35. New cstate: current_term: 35 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:40.934762 3722 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 38 LEADER]: Stepping down as leader of term 38
I20250112 20:51:40.934906 3722 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.935204 3722 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.24, Last appended by leader: 24, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.935503 3722 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 38 FOLLOWER]: Advancing to term 39
W20250112 20:51:40.935487 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.936734 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.936949 3374 raft_consensus.cc:1212] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 LEADER]: Deduplicated request from leader. Original: 36.17->[36.18-38.22] Dedup: 38.22->[]
I20250112 20:51:40.937062 3511 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 38. Current term is 39. Ops: [38.23-38.23]
I20250112 20:51:40.937144 3374 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 LEADER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 38. Current term is 39. Ops: [36.18-38.22]
I20250112 20:51:40.937472 3510 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.937611 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Advancing to term 40
I20250112 20:51:40.939671 3510 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 40 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.940338 3510 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 40 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
W20250112 20:51:40.941147 3766 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: Replica 710959530a634cc991544c14efad9e96 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 39 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:40.941457 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:40.941639 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 35 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.941769 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 35 FOLLOWER]: Advancing to term 36
I20250112 20:51:40.942039 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.942245 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 39 FOLLOWER]: Advancing to term 40
I20250112 20:51:40.943953 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.944692 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 40 because replica has last-logged OpId of term: 38 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 22.
I20250112 20:51:40.944716 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 36 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:40.945184 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:40.945168 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 40 candidate_status { last_received { term: 38 index: 22 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.945396 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 35 FOLLOWER]: Advancing to term 36
I20250112 20:51:40.945354 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 36 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.945583 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 LEADER]: Stepping down as leader of term 35
I20250112 20:51:40.945699 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.945956 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.29, Last appended by leader: 29, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.946238 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 35 FOLLOWER]: Advancing to term 36
W20250112 20:51:40.947525 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.945418 3374 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 LEADER]: Stepping down as leader of term 39
I20250112 20:51:40.947995 3374 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:40.948071 3650 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 36.
I20250112 20:51:40.948441 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.948364 3374 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 39.23, Last appended by leader: 23, Current term: 39, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.948675 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 FOLLOWER]: Advancing to term 40
I20250112 20:51:40.948756 3746 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Leader election won for term 36
I20250112 20:51:40.949033 3746 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 36 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.949326 3746 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 33.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.950539 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.951545 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 40 because replica has last-logged OpId of term: 39 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 22.
I20250112 20:51:40.951975 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 40 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:40.952132 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 36 because replica has last-logged OpId of term: 35 index: 29, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 24.
I20250112 20:51:40.952225 3746 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 40 FOLLOWER]: Leader election lost for term 40. Reason: could not achieve majority
I20250112 20:51:40.952000 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 35 to 36, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 36 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:40.956185 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:40.956997 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.966095 3375 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 35 index: 29. Preceding OpId from leader: term: 36 index: 26. (term mismatch)
I20250112 20:51:40.966229 3375 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 25
I20250112 20:51:40.966324 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 33.26
I20250112 20:51:40.966518 3650 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 36 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
W20250112 20:51:40.966536 3375 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
I20250112 20:51:40.967115 3819 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250112 20:51:40.967170 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 33.27
W20250112 20:51:40.967339 3375 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
W20250112 20:51:40.967921 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.967984 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 33.28
W20250112 20:51:40.968171 3375 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
I20250112 20:51:40.968503 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 35.29
I20250112 20:51:40.968632 3375 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:40.969256 3819 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20250112 20:51:40.969442 3819 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 33.25 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 24
I20250112 20:51:40.970273 3375 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 24
I20250112 20:51:40.970397 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 33.25
W20250112 20:51:40.970615 3375 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
W20250112 20:51:40.981212 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.985332 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:40.986048 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.986268 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 40 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.986403 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 40 FOLLOWER]: Advancing to term 41
W20250112 20:51:40.989655 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:40.989751 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 41 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:40.989886 3606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:40.990552 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 41 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:40.991235 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:40.991472 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:40.991544 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 40 FOLLOWER]: Advancing to term 41
I20250112 20:51:40.991704 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 36 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:40.991811 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 36 FOLLOWER]: Advancing to term 37
I20250112 20:51:40.992596 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 41 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:40.992818 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 40 FOLLOWER]: Advancing to term 41
I20250112 20:51:40.994681 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 41.
I20250112 20:51:40.995093 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:40.995760 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.995980 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 41 FOLLOWER]: Leader election won for term 41
I20250112 20:51:40.996054 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 41 because replica has last-logged OpId of term: 39 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 24.
I20250112 20:51:40.996467 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 41 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:40.996634 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 37 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:40.996902 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.24, Last appended by leader: 24, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:40.998643 3650 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Rejecting Update request from peer f15597f529254479b5987d2613686dfb for earlier term 36. Current term is 37. Ops: [36.30-36.34]
I20250112 20:51:40.999240 3761 consensus_queue.cc:1046] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: INVALID_TERM, Last received: 36.29, Next index: 30, Last known committed idx: 26, Time since last communication: 0.000s
I20250112 20:51:40.999476 3375 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 39 index: 23. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20250112 20:51:40.999568 3761 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 36 LEADER]: Stepping down as leader of term 36
I20250112 20:51:40.999657 3761 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 36 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:40.999965 3814 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 17, Time since last communication: 0.000s
I20250112 20:51:40.999951 3761 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 36.34, Last appended by leader: 34, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.000119 3814 consensus_queue.cc:1225] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 39.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20250112 20:51:41.000280 3761 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 36 FOLLOWER]: Advancing to term 37
I20250112 20:51:41.000072 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 39 to 41, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 41 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:41.001370 3373 raft_consensus.cc:1212] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Deduplicated request from leader. Original: 36.17->[36.18-41.26] Dedup: 38.22->[38.23-41.26]
I20250112 20:51:41.001539 3373 pending_rounds.cc:77] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 22
I20250112 20:51:41.001605 3373 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 39.23
I20250112 20:51:41.001693 3373 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:41.006875 3511 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 41 index: 26. (index mismatch)
I20250112 20:51:41.008116 3731 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 17, Time since last communication: 0.000s
I20250112 20:51:41.008531 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 37 candidate_status { last_received { term: 36 index: 29 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
W20250112 20:51:41.008594 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.008783 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 36 FOLLOWER]: Advancing to term 37
I20250112 20:51:41.011426 3374 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 37 because replica has last-logged OpId of term: 36 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 29.
I20250112 20:51:41.011799 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 37 candidate_status { last_received { term: 36 index: 29 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.012054 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 37 because replica has last-logged OpId of term: 36 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 29.
I20250112 20:51:41.015610 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:41.019673 3731 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
W20250112 20:51:41.019721 3328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.030828 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.033360 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.033439 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:41.034304 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
W20250112 20:51:41.037397 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.035923 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.039914 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Advancing to term 42
W20250112 20:51:41.041026 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.042940 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.043394 3372 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 41. Current term is 42. Ops: [41.32-41.32]
W20250112 20:51:41.043438 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.044139 3722 consensus_queue.cc:1046] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: INVALID_TERM, Last received: 41.31, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 20:51:41.044412 3722 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 41 LEADER]: Stepping down as leader of term 41
I20250112 20:51:41.044530 3722 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.044804 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 42 candidate_status { last_received { term: 41 index: 31 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.044862 3722 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 41.34, Last appended by leader: 34, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.045246 3722 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 41 FOLLOWER]: Advancing to term 42
W20250112 20:51:41.045639 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.046393 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 42 candidate_status { last_received { term: 41 index: 31 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.046703 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 42 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
W20250112 20:51:41.047341 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.048417 3372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:41.048650 3372 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.048763 3372 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 37 FOLLOWER]: Advancing to term 38
I20250112 20:51:41.050767 3372 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.051282 3372 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 38 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.051980 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 38 candidate_status { last_received { term: 36 index: 34 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.052217 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 37 FOLLOWER]: Advancing to term 38
W20250112 20:51:41.052407 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.053522 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
W20250112 20:51:41.053653 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.054246 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 42 because replica has last-logged OpId of term: 41 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 31.
I20250112 20:51:41.055051 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 38.
W20250112 20:51:41.055572 3328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.056134 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb
I20250112 20:51:41.056448 3812 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20250112 20:51:41.056644 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:41.057053 3812 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 FOLLOWER]: Leader election won for term 38
I20250112 20:51:41.057217 3812 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.057301 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:41.057507 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.057462 3812 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 36.34, Last appended by leader: 34, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.057636 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Advancing to term 42
I20250112 20:51:41.059931 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 37 FOLLOWER]: Advancing to term 38
I20250112 20:51:41.059983 3652 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 38 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 36 index: 29. Preceding OpId from leader: term: 38 index: 36. (index mismatch)
I20250112 20:51:41.060681 3812 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 26, Time since last communication: 0.000s
W20250112 20:51:41.061928 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.060391 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 42 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.062111 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 36 to 38, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 38 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:41.062866 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 42 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.063989 3509 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 38 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 36 index: 34. Preceding OpId from leader: term: 38 index: 36. (index mismatch)
I20250112 20:51:41.064893 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 42 candidate_status { last_received { term: 41 index: 31 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.065146 3375 raft_consensus.cc:2388] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb in current term 42: Already voted for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in this term.
I20250112 20:51:41.068229 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:41.068207 3812 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 20:51:41.068450 3509 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 38 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.068601 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 38 FOLLOWER]: Advancing to term 39
I20250112 20:51:41.071287 3509 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.071864 3509 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 39 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.072258 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 39 candidate_status { last_received { term: 36 index: 34 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.072453 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 LEADER]: Stepping down as leader of term 38
I20250112 20:51:41.072604 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.072948 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 38.37, Last appended by leader: 37, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.073637 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 42 candidate_status { last_received { term: 41 index: 31 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.073834 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 39 candidate_status { last_received { term: 36 index: 34 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.073974 3650 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 42 because replica has last-logged OpId of term: 41 index: 34, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 31.
I20250112 20:51:41.074052 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 38 FOLLOWER]: Advancing to term 39
W20250112 20:51:41.074324 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.074395 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 38 FOLLOWER]: Advancing to term 39
I20250112 20:51:41.075093 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 42 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.075392 3720 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 42 FOLLOWER]: Leader election lost for term 42. Reason: could not achieve majority
I20250112 20:51:41.076247 3509 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 38. Current term is 39. Ops: [38.35-38.36]
W20250112 20:51:41.076715 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.076877 3651 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 39 because replica has last-logged OpId of term: 38 index: 36, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 34.
I20250112 20:51:41.077168 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 38 candidate_status { last_received { term: 36 index: 34 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.077250 3652 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 39 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 38. Current term is 39. Ops: [38.37-38.37]
I20250112 20:51:41.077400 3510 raft_consensus.cc:2363] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 38. Current term is 39.
I20250112 20:51:41.077948 3303 leader_election.cc:400] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 38 election: Vote denied by peer f15597f529254479b5987d2613686dfb (127.0.104.2:44825) with higher term. Message: Invalid argument: T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 38. Current term is 39.
I20250112 20:51:41.078147 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 39 because replica has last-logged OpId of term: 38 index: 37, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 34.
I20250112 20:51:41.078488 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.078773 3720 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
W20250112 20:51:41.086858 3328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.089776 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.099671 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.102706 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.109759 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
W20250112 20:51:41.110495 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.110533 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:41.110752 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.110868 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Advancing to term 43
I20250112 20:51:41.112845 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 43 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.113488 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 43 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
W20250112 20:51:41.113647 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.113898 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 43 candidate_status { last_received { term: 41 index: 34 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.114005 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 43 candidate_status { last_received { term: 41 index: 34 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.114109 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 42 FOLLOWER]: Advancing to term 43
I20250112 20:51:41.114243 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Advancing to term 43
I20250112 20:51:41.114701 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:41.114882 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 39 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.115006 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 39 FOLLOWER]: Advancing to term 40
I20250112 20:51:41.116190 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 43.
I20250112 20:51:41.116680 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 43 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:41.116986 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 43 FOLLOWER]: Leader election won for term 43
I20250112 20:51:41.117067 3375 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 43 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 43.
I20250112 20:51:41.117147 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 43 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.117436 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 41.34, Last appended by leader: 34, Current term: 43, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.117808 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 40 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.118373 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 40 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:41.118844 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 40 candidate_status { last_received { term: 38 index: 36 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.119065 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 39 FOLLOWER]: Advancing to term 40
I20250112 20:51:41.119352 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 40 candidate_status { last_received { term: 38 index: 36 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.119540 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 39 FOLLOWER]: Advancing to term 40
I20250112 20:51:41.121268 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 40 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 40 because replica has last-logged OpId of term: 38 index: 37, which is greater than that of the candidate, which has last-logged OpId of term: 38 index: 36.
I20250112 20:51:41.120994 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 41 to 43. New cstate: current_term: 43 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:41.122069 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 40.
I20250112 20:51:41.122545 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.122845 3722 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 40 FOLLOWER]: Leader election won for term 40
I20250112 20:51:41.123081 3722 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 40 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.123349 3722 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 38.36, Last appended by leader: 36, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.126096 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 38 to 40, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 40 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:41.127831 3328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.131523 3510 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 40 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 36 index: 34. Preceding OpId from leader: term: 40 index: 38. (index mismatch)
I20250112 20:51:41.131626 3375 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 40 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 38 index: 37. Preceding OpId from leader: term: 40 index: 38. (index mismatch)
I20250112 20:51:41.132061 3731 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 33, Time since last communication: 0.000s
I20250112 20:51:41.132318 3722 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250112 20:51:41.132524 3722 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 38.37 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 36
I20250112 20:51:41.133738 3375 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 36
I20250112 20:51:41.133862 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 38.37
W20250112 20:51:41.134020 3375 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
I20250112 20:51:41.135890 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:41.136718 3372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:41.136912 3372 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 43 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.137038 3372 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 43 FOLLOWER]: Advancing to term 44
I20250112 20:51:41.139839 3372 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.141366 3372 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 44 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
W20250112 20:51:41.141769 3818 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.142542 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 44 candidate_status { last_received { term: 41 index: 31 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.143118 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 44 candidate_status { last_received { term: 41 index: 31 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.143194 3372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:41.143339 3652 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 43 LEADER]: Stepping down as leader of term 43
I20250112 20:51:41.143393 3372 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 40 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.143425 3652 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 43 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.143524 3372 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 40 FOLLOWER]: Advancing to term 41
I20250112 20:51:41.143723 3652 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 43.35, Last appended by leader: 35, Current term: 43, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.144027 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 43 FOLLOWER]: Advancing to term 44
I20250112 20:51:41.146231 3372 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.147634 3372 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 41 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.148196 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 41 candidate_status { last_received { term: 40 index: 38 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.148388 3651 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 40 LEADER]: Stepping down as leader of term 40
I20250112 20:51:41.148494 3651 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 40 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.148801 3651 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 40.38, Last appended by leader: 38, Current term: 40, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.149173 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 40 FOLLOWER]: Advancing to term 41
I20250112 20:51:41.148571 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 43 FOLLOWER]: Advancing to term 44
I20250112 20:51:41.149688 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 44 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 44 because replica has last-logged OpId of term: 43 index: 35, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 31.
I20250112 20:51:41.151945 3651 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 41.
I20250112 20:51:41.152314 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 44.
I20250112 20:51:41.152356 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:41.152760 3816 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Leader election won for term 41
W20250112 20:51:41.152839 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.152827 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96
I20250112 20:51:41.152947 3816 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.153539 3812 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 FOLLOWER]: Leader election won for term 44
I20250112 20:51:41.147753 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 41 candidate_status { last_received { term: 40 index: 38 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.153673 3812 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.153820 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 40 FOLLOWER]: Advancing to term 41
I20250112 20:51:41.154199 3812 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 41.31, Last appended by leader: 31, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.154201 3816 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 40.38, Last appended by leader: 38, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.155892 3509 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 41.
I20250112 20:51:41.156766 3209 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 40 to 41, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 41 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:41.158210 3606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.158826 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.160982 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.161441 3802 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.162932 3209 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 43 to 44, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 44 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:41.170137 3509 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 40 index: 38. Preceding OpId from leader: term: 41 index: 40. (index mismatch)
W20250112 20:51:41.170311 3802 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.170542 3651 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 41 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 40 index: 38. Preceding OpId from leader: term: 41 index: 40. (index mismatch)
I20250112 20:51:41.170711 3812 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20250112 20:51:41.171061 3816 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
W20250112 20:51:41.182416 3606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.189584 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.192327 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.202008 3510 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 41 index: 31. Preceding OpId from leader: term: 44 index: 33. (index mismatch)
I20250112 20:51:41.202421 3652 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 44 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 43 index: 35. Preceding OpId from leader: term: 44 index: 33. (term mismatch)
I20250112 20:51:41.202559 3652 pending_rounds.cc:77] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting all ops after (but not including) 32
I20250112 20:51:41.202605 3816 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 20:51:41.202713 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 41.33
W20250112 20:51:41.202941 3652 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:41.203676 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 41.34
W20250112 20:51:41.203876 3652 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:41.204439 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted NO_OP operation due to leader change: 43.35
I20250112 20:51:41.204574 3652 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 44 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:41.205336 3816 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250112 20:51:41.205510 3816 consensus_queue.cc:1225] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer 710959530a634cc991544c14efad9e96 log is divergent from this leader: its last log entry 41.32 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 31
I20250112 20:51:41.209230 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:41.214910 3823 mvcc.cc:204] Tried to move back new op lower bound from 7113585054528294912 to 7113585054329868288. Current Snapshot: MvccSnapshot[applied={T|T < 7113585054528294912 or (T in {7113585054528294912})}]
I20250112 20:51:41.216945 3651 pending_rounds.cc:77] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting all ops after (but not including) 31
I20250112 20:51:41.217070 3651 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 41.32
I20250112 20:51:41.219825 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:41.220045 3510 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.220149 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Advancing to term 45
W20250112 20:51:41.217275 3651 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:41.226845 3510 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.239868 3511 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 44. Current term is 45. Ops: [44.37-44.37]
I20250112 20:51:41.240267 3510 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 45 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.241920 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:41.241931 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 45 candidate_status { last_received { term: 44 index: 36 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.242134 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.242178 3374 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 LEADER]: Stepping down as leader of term 44
I20250112 20:51:41.242283 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 41 FOLLOWER]: Advancing to term 42
I20250112 20:51:41.242296 3374 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.242612 3374 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 44.41, Last appended by leader: 41, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.242889 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 FOLLOWER]: Advancing to term 45
I20250112 20:51:41.246132 3374 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 45 because replica has last-logged OpId of term: 44 index: 41, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 36.
I20250112 20:51:41.247766 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 42 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.248449 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 42 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.248401 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 45 candidate_status { last_received { term: 44 index: 36 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.248626 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 44 FOLLOWER]: Advancing to term 45
I20250112 20:51:41.250047 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:41.250923 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 45 because replica has last-logged OpId of term: 44 index: 40, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 36.
I20250112 20:51:41.250923 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:41.251143 3651 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 45 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.251272 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 45 FOLLOWER]: Advancing to term 46
I20250112 20:51:41.252671 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.253104 3746 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
I20250112 20:51:41.253400 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 42 candidate_status { last_received { term: 41 index: 52 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.253614 3374 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 LEADER]: Stepping down as leader of term 41
I20250112 20:51:41.253723 3374 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.254048 3374 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 41.52, Last appended by leader: 52, Current term: 41, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.254478 3374 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 41 FOLLOWER]: Advancing to term 42
I20250112 20:51:41.254875 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 42 candidate_status { last_received { term: 41 index: 52 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.255051 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 41 FOLLOWER]: Advancing to term 42
I20250112 20:51:41.257128 3374 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 42.
I20250112 20:51:41.257582 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 42 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:41.258047 3651 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 46 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.258381 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.259115 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 42.
I20250112 20:51:41.259433 3720 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 42 FOLLOWER]: Leader election won for term 42
I20250112 20:51:41.259605 3720 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 42 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:41.259879 3720 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 41.52, Last appended by leader: 52, Current term: 42, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.260454 3825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.264029 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 41 to 42, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 42 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:41.267460 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 46 candidate_status { last_received { term: 44 index: 40 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.267719 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Advancing to term 46
I20250112 20:51:41.268157 3375 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 41 index: 52. Preceding OpId from leader: term: 42 index: 54. (index mismatch)
W20250112 20:51:41.270996 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.270972 3761 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250112 20:51:41.273880 3651 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 46 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:41.274338 3652 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 41 index: 52. Preceding OpId from leader: term: 42 index: 54. (index mismatch)
I20250112 20:51:41.275152 3761 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250112 20:51:41.275424 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 46.
I20250112 20:51:41.276412 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:41.276695 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.276723 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:41.276966 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 42 FOLLOWER]: Advancing to term 43
I20250112 20:51:41.277055 3722 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 46 FOLLOWER]: Leader election won for term 46
I20250112 20:51:41.277230 3722 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 46 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.277527 3722 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 44.40, Last appended by leader: 39, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.280340 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 44 to 46, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 46 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:41.283211 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 46 candidate_status { last_received { term: 44 index: 40 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.283452 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 45 FOLLOWER]: Advancing to term 46
W20250112 20:51:41.286361 3825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.287606 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 46 because replica has last-logged OpId of term: 44 index: 41, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 40.
I20250112 20:51:41.288978 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 43 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.289441 3651 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 43 FOLLOWER]: Rejecting Update request from peer f15597f529254479b5987d2613686dfb for earlier term 42. Current term is 43. Ops: [42.53-42.54]
I20250112 20:51:41.290210 3761 consensus_queue.cc:1046] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: INVALID_TERM, Last received: 41.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250112 20:51:41.290584 3761 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 42 LEADER]: Stepping down as leader of term 42
I20250112 20:51:41.290547 3372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 43 candidate_status { last_received { term: 41 index: 52 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.290699 3761 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 42 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:41.290719 3372 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 42 FOLLOWER]: Advancing to term 43
W20250112 20:51:41.291015 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.290957 3761 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 42.57, Last appended by leader: 57, Current term: 42, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.291303 3761 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 42 FOLLOWER]: Advancing to term 43
I20250112 20:51:41.293437 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 43 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
W20250112 20:51:41.293471 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.294839 3355 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.294939 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.295663 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.296574 3372 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 43 because replica has last-logged OpId of term: 42 index: 57, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 52.
I20250112 20:51:41.296861 3375 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 43 FOLLOWER]: Rejecting Update request from peer f15597f529254479b5987d2613686dfb for earlier term 42. Current term is 43. Ops: []
I20250112 20:51:41.296782 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 43 candidate_status { last_received { term: 41 index: 52 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.297091 3510 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 43 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 43 because replica has last-logged OpId of term: 42 index: 57, which is greater than that of the candidate, which has last-logged OpId of term: 41 index: 52.
I20250112 20:51:41.297717 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 43 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
W20250112 20:51:41.298007 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.298022 3722 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 43 FOLLOWER]: Leader election lost for term 43. Reason: could not achieve majority
W20250112 20:51:41.298789 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.301508 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.305310 3632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.305330 3631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.307647 3510 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 46 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 44 index: 36. Preceding OpId from leader: term: 46 index: 42. (index mismatch)
I20250112 20:51:41.307855 3374 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 44 index: 41. Preceding OpId from leader: term: 46 index: 42. (index mismatch)
I20250112 20:51:41.308415 3722 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 36, Time since last communication: 0.000s
I20250112 20:51:41.308653 3722 consensus_queue.cc:1225] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 44.41 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 36
I20250112 20:51:41.308876 3814 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 41, Last known committed idx: 35, Time since last communication: 0.000s
I20250112 20:51:41.310262 3374 raft_consensus.cc:1212] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Deduplicated request from leader. Original: 44.36->[44.37-46.42] Dedup: 44.40->[46.41-46.42]
I20250112 20:51:41.310868 3374 pending_rounds.cc:77] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 40
I20250112 20:51:41.310979 3374 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 44.41
W20250112 20:51:41.311206 3374 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
W20250112 20:51:41.311301 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.321436 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.325057 3327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.325330 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:41.326486 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
W20250112 20:51:41.332268 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.332384 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.334697 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.335240 3802 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.338089 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.340273 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.342500 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.342638 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Advancing to term 47
W20250112 20:51:41.344105 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.345253 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.346493 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 47 candidate_status { last_received { term: 46 index: 45 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.347750 3372 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 46. Current term is 47. Ops: [46.46-46.48]
W20250112 20:51:41.348804 3632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.349284 3631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.349592 3731 consensus_queue.cc:1046] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: INVALID_TERM, Last received: 46.45, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
I20250112 20:51:41.349817 3731 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 46 LEADER]: Stepping down as leader of term 46
I20250112 20:51:41.349900 3731 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.350134 3731 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 46.48, Last appended by leader: 48, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.350462 3731 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 46 FOLLOWER]: Advancing to term 47
W20250112 20:51:41.350582 3825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.351322 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 47 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.352245 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:41.352399 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 43 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.352537 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 43 FOLLOWER]: Advancing to term 44
I20250112 20:51:41.355243 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.356061 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.358429 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 47 candidate_status { last_received { term: 46 index: 45 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.358824 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 47 because replica has last-logged OpId of term: 46 index: 48, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 45.
I20250112 20:51:41.360183 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 47 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb
I20250112 20:51:41.360392 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 44 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.360466 3721 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Leader election lost for term 47. Reason: could not achieve majority
W20250112 20:51:41.362035 3632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.363459 3631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.366596 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 44 candidate_status { last_received { term: 42 index: 57 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.366884 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 43 FOLLOWER]: Advancing to term 44
W20250112 20:51:41.368031 3802 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.369855 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.370416 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.359884 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.365151 3490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.366593 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.368697 3486 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.356915 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 44 candidate_status { last_received { term: 42 index: 57 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.374250 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 43 FOLLOWER]: Advancing to term 44
W20250112 20:51:41.371091 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.372901 3652 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 44.
I20250112 20:51:41.377226 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 44 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
W20250112 20:51:41.377264 3327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.377812 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 44.
I20250112 20:51:41.378156 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 FOLLOWER]: Leader election won for term 44
I20250112 20:51:41.378319 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
W20250112 20:51:41.378499 3327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Service unavailable: leader is not yet ready
I20250112 20:51:41.378607 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 55, Committed index: 55, Last appended: 42.57, Last appended by leader: 57, Current term: 44, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.380153 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.380841 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.382161 3651 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 44 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 41 index: 52. Preceding OpId from leader: term: 44 index: 59. (index mismatch)
I20250112 20:51:41.382707 3769 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 52, Time since last communication: 0.000s
I20250112 20:51:41.382556 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 42 to 44, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 44 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:41.390337 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.391849 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.392110 3510 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 42 index: 57. Preceding OpId from leader: term: 44 index: 59. (index mismatch)
I20250112 20:51:41.394042 3721 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
W20250112 20:51:41.394874 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.398752 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.398880 3825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.402292 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:41.403097 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:41.403285 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 46 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.403401 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 46 FOLLOWER]: Advancing to term 47
W20250112 20:51:41.403640 3327 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.406242 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 47 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.406963 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 47 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.407820 3509 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
W20250112 20:51:41.408089 3631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.409505 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 47 candidate_status { last_received { term: 46 index: 48 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.411067 3372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 47 candidate_status { last_received { term: 46 index: 48 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.411319 3372 raft_consensus.cc:2388] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb in current term 47: Already voted for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in this term.
I20250112 20:51:41.412324 3650 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 47.
I20250112 20:51:41.413290 3439 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.413818 3746 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 47 FOLLOWER]: Leader election won for term 47
I20250112 20:51:41.414009 3746 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 47 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:41.414386 3746 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 46.48, Last appended by leader: 48, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.421449 3652 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 46 index: 48. Preceding OpId from leader: term: 47 index: 50. (index mismatch)
I20250112 20:51:41.423378 3746 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20250112 20:51:41.425359 3372 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 46 index: 45. Preceding OpId from leader: term: 47 index: 50. (index mismatch)
W20250112 20:51:41.425554 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.428309 3746 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 44, Time since last communication: 0.000s
I20250112 20:51:41.429421 3209 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 46 to 47, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 47 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
W20250112 20:51:41.467077 3805 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.468649 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.474694 3509 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.474875 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 44 FOLLOWER]: Advancing to term 45
I20250112 20:51:41.477670 3509 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.481767 3509 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 45 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.485652 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 45 candidate_status { last_received { term: 44 index: 63 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.490199 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 44 FOLLOWER]: Advancing to term 45
I20250112 20:51:41.490586 3372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 45 candidate_status { last_received { term: 44 index: 63 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.490883 3372 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 LEADER]: Stepping down as leader of term 44
I20250112 20:51:41.490983 3372 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.491294 3372 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 44.75, Last appended by leader: 75, Current term: 44, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.491556 3372 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 44 FOLLOWER]: Advancing to term 45
I20250112 20:51:41.494190 3651 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 45 because replica has last-logged OpId of term: 44 index: 73, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 63.
I20250112 20:51:41.494573 3372 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 45 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 45 because replica has last-logged OpId of term: 44 index: 75, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 63.
I20250112 20:51:41.496078 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 45 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.497373 3652 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 45 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 44. Current term is 45. Ops: [44.74-44.75]
W20250112 20:51:41.504726 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.508606 3355 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.509061 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.509351 3510 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 44. Current term is 45. Ops: [44.64-44.75]
I20250112 20:51:41.510738 3819 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Leader election lost for term 45. Reason: could not achieve majority
W20250112 20:51:41.522178 3486 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.523458 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.524366 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.524960 3488 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.530501 3825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.538903 3631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.540664 3631 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.540999 3632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.544521 3632 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.549360 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.551448 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:41.552785 3650 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
W20250112 20:51:41.552922 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.553014 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.555027 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.555655 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.556000 3825 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.556623 3465 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.559185 3650 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.559356 3650 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Advancing to term 48
W20250112 20:51:41.560979 3465 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.563253 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.565094 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.568764 3355 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.568995 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.571144 3650 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.572484 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 48 candidate_status { last_received { term: 47 index: 64 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.572630 3650 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 48 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:41.572731 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Advancing to term 48
I20250112 20:51:41.573987 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:41.574164 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 45 FOLLOWER]: Starting forced leader election (received explicit request)
W20250112 20:51:41.574179 3490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.574275 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 45 FOLLOWER]: Advancing to term 46
I20250112 20:51:41.575201 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 48 candidate_status { last_received { term: 47 index: 64 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.575395 3510 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 47 LEADER]: Stepping down as leader of term 47
I20250112 20:51:41.575493 3510 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 47 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:41.575548 3375 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 48 because replica has last-logged OpId of term: 47 index: 67, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 64.
W20250112 20:51:41.576125 3490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.576341 3510 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 47.67, Last appended by leader: 67, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.576712 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 47 FOLLOWER]: Advancing to term 48
W20250112 20:51:41.576790 3490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.576833 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 46 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.577486 3490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.578037 3509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 46 candidate_status { last_received { term: 44 index: 73 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.578234 3509 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 45 FOLLOWER]: Advancing to term 46
I20250112 20:51:41.578222 3372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 46 candidate_status { last_received { term: 44 index: 73 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.578419 3372 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 45 FOLLOWER]: Advancing to term 46
I20250112 20:51:41.580216 3510 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 48 because replica has last-logged OpId of term: 47 index: 67, which is greater than that of the candidate, which has last-logged OpId of term: 47 index: 64.
W20250112 20:51:41.580971 3817 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: Replica f15597f529254479b5987d2613686dfb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 48 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
I20250112 20:51:41.581053 3372 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 46 because replica has last-logged OpId of term: 44 index: 75, which is greater than that of the candidate, which has last-logged OpId of term: 44 index: 73.
I20250112 20:51:41.581512 3509 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 46 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 46.
I20250112 20:51:41.582227 3580 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 710959530a634cc991544c14efad9e96; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb
I20250112 20:51:41.582556 3722 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20250112 20:51:41.582755 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 46 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.583047 3722 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 46 FOLLOWER]: Leader election won for term 46
I20250112 20:51:41.583194 3722 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 46 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
W20250112 20:51:41.583283 3785 tablet_replica.cc:1307] Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 28 similar messages]
W20250112 20:51:41.583436 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.583457 3722 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 72, Committed index: 72, Last appended: 44.73, Last appended by leader: 73, Current term: 46, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.585923 3490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.586195 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 46 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:41.587152 3212 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 44 to 46, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 46 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:41.591003 3372 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 44 index: 75. Preceding OpId from leader: term: 46 index: 75. (term mismatch)
I20250112 20:51:41.591183 3372 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 74
I20250112 20:51:41.591329 3372 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 44.75
W20250112 20:51:41.591548 3372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Aborted: Op aborted by new leader
I20250112 20:51:41.592170 3509 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 46 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 44 index: 63. Preceding OpId from leader: term: 46 index: 75. (index mismatch)
I20250112 20:51:41.592819 3722 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 59, Time since last communication: 0.000s
W20250112 20:51:41.593014 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.593703 3722 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20250112 20:51:41.593868 3722 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 44.74 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 73
I20250112 20:51:41.596036 3372 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 73
I20250112 20:51:41.596112 3372 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 44.74
W20250112 20:51:41.596262 3372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Aborted: Op aborted by new leader
I20250112 20:51:41.600080 3822 mvcc.cc:204] Tried to move back new op lower bound from 7113585056120299520 to 7113585056088281088. Current Snapshot: MvccSnapshot[applied={T|T < 7113585055256436736 or (T in {7113585055266459648,7113585055283240960,7113585055382032384,7113585055411253248,7113585055445024768,7113585055447117824,7113585055450038272,7113585055487459328,7113585055489863680,7113585055537864704,7113585055270555648})}]
W20250112 20:51:41.609019 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.625715 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.627673 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.628456 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:41.629262 3374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:41.629468 3374 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 48 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.629602 3374 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 48 FOLLOWER]: Advancing to term 49
I20250112 20:51:41.632397 3374 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.633095 3374 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 49 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
W20250112 20:51:41.635242 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.635905 3355 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.637446 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 49 candidate_status { last_received { term: 47 index: 67 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.637688 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Advancing to term 49
W20250112 20:51:41.638295 3466 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.641113 3651 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 49.
I20250112 20:51:41.643642 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
W20250112 20:51:41.643865 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.644016 3721 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Leader election won for term 49
I20250112 20:51:41.644217 3721 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
W20250112 20:51:41.644484 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.644574 3721 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 67, Committed index: 67, Last appended: 47.67, Last appended by leader: 67, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.645648 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.651571 3209 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 47 to 49, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 49 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:41.653024 3372 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:41.653532 3509 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 48 FOLLOWER]: Advancing to term 49
W20250112 20:51:41.653589 3488 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.656662 3509 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 49 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 47 index: 67. Preceding OpId from leader: term: 49 index: 69. (index mismatch)
W20250112 20:51:41.656894 3488 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.657183 3721 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 67, Time since last communication: 0.000s
I20250112 20:51:41.657238 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 49 candidate_status { last_received { term: 47 index: 67 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.657881 3651 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 49 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 47 index: 64. Preceding OpId from leader: term: 49 index: 69. (index mismatch)
I20250112 20:51:41.661625 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 49.
I20250112 20:51:41.661633 3721 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 68, Last known committed idx: 64, Time since last communication: 0.000s
I20250112 20:51:41.662595 3372 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.662731 3372 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 46 FOLLOWER]: Advancing to term 47
I20250112 20:51:41.665659 3372 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.666826 3372 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 47 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.670239 3650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 47 candidate_status { last_received { term: 46 index: 97 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.670471 3650 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 46 LEADER]: Stepping down as leader of term 46
I20250112 20:51:41.670578 3650 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 46 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.670912 3650 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 46.98, Last appended by leader: 98, Current term: 46, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.671140 3650 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 46 FOLLOWER]: Advancing to term 47
I20250112 20:51:41.673913 3372 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 46. Current term is 47. Ops: [46.98-46.98]
I20250112 20:51:41.674440 3650 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 47 because replica has last-logged OpId of term: 46 index: 98, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 97.
W20250112 20:51:41.678200 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.679589 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.681118 3606 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.681593 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.682490 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 47 candidate_status { last_received { term: 46 index: 97 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.682755 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 46 FOLLOWER]: Advancing to term 47
W20250112 20:51:41.682773 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.684243 3490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.684402 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.685222 3486 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.685794 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 47 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 47.
W20250112 20:51:41.686429 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.686661 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 47 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96
I20250112 20:51:41.687534 3812 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Leader election won for term 47
I20250112 20:51:41.687687 3812 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.687952 3812 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 46.97, Last appended by leader: 97, Current term: 47, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.695139 3209 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 46 to 47, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 47 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:41.704972 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:41.705811 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:41.706055 3510 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 49 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.706722 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 49 FOLLOWER]: Advancing to term 50
I20250112 20:51:41.710180 3510 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.710618 3511 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 49. Current term is 50. Ops: [49.74-49.74]
I20250112 20:51:41.711542 3721 consensus_queue.cc:1046] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: INVALID_TERM, Last received: 49.73, Next index: 74, Last known committed idx: 73, Time since last communication: 0.001s
I20250112 20:51:41.712347 3511 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 47 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 46 index: 97. Preceding OpId from leader: term: 47 index: 99. (index mismatch)
I20250112 20:51:41.712349 3652 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 46 index: 98. Preceding OpId from leader: term: 47 index: 99. (index mismatch)
I20250112 20:51:41.712961 3510 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 50 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.713392 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 50 candidate_status { last_received { term: 49 index: 73 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:41.713591 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 49 FOLLOWER]: Advancing to term 50
I20250112 20:51:41.714951 3721 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 LEADER]: Stepping down as leader of term 49
I20250112 20:51:41.715067 3721 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.716434 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 50 because replica has last-logged OpId of term: 49 index: 74, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 73.
I20250112 20:51:41.716653 3769 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.004s
I20250112 20:51:41.716977 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:41.717130 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 47 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.717242 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 47 FOLLOWER]: Advancing to term 48
I20250112 20:51:41.719437 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 48 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
W20250112 20:51:41.719561 3789 tablet_replica.cc:1307] Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 30 similar messages]
W20250112 20:51:41.719799 3789 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.719563 3796 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.720077 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 48 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:41.720302 3769 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer 710959530a634cc991544c14efad9e96 log is divergent from this leader: its last log entry 46.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
W20250112 20:51:41.721340 3355 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.721707 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 48 candidate_status { last_received { term: 46 index: 97 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
W20250112 20:51:41.721904 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.721941 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 47 FOLLOWER]: Advancing to term 48
I20250112 20:51:41.722530 3374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 50 candidate_status { last_received { term: 49 index: 73 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.723039 3812 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 92, Time since last communication: 0.002s
I20250112 20:51:41.723479 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 50 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.723948 3761 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Leader election lost for term 50. Reason: could not achieve majority
I20250112 20:51:41.724575 3651 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 48 because replica has last-logged OpId of term: 46 index: 98, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 97.
I20250112 20:51:41.724838 3652 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 47. Current term is 48. Ops: [47.98-47.99]
W20250112 20:51:41.725935 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.726738 3510 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 48 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 47. Current term is 48. Ops: [47.98-47.100]
W20250112 20:51:41.727017 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.725090 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 48 candidate_status { last_received { term: 46 index: 97 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.727927 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 48 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.728274 3761 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 48 FOLLOWER]: Leader election lost for term 48. Reason: could not achieve majority
I20250112 20:51:41.728530 3769 consensus_queue.cc:1046] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.003s
I20250112 20:51:41.728498 3721 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 49.74, Last appended by leader: 74, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.728744 3769 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer 710959530a634cc991544c14efad9e96 log is divergent from this leader: its last log entry 46.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
I20250112 20:51:41.728871 3721 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Advancing to term 50
I20250112 20:51:41.729569 3812 consensus_queue.cc:1046] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: INVALID_TERM, Last received: 46.97, Next index: 98, Last known committed idx: 92, Time since last communication: 0.000s
W20250112 20:51:41.731848 3768 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: Replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 50 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } } }
W20250112 20:51:41.731972 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.731992 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.732091 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.732697 3816 consensus_queue.cc:1159] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "710959530a634cc991544c14efad9e96" responder_term: 49 status { last_received { term: 49 index: 74 } last_committed_idx: 74 last_received_current_leader { term: 49 index: 74 } }
W20250112 20:51:41.732827 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.734293 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.734746 3769 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 LEADER]: Stepping down as leader of term 47
I20250112 20:51:41.734889 3769 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:41.735252 3769 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 47.102, Last appended by leader: 102, Current term: 47, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.736035 3652 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 47. Current term is 48. Ops: [47.98-47.102]
I20250112 20:51:41.736712 3510 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 48 FOLLOWER]: Rejecting Update request from peer d3f9c3f8fe0d4459b54d92f75a2c84b7 for earlier term 47. Current term is 48. Ops: [47.98-47.102]
I20250112 20:51:41.736842 3812 consensus_queue.cc:1046] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: INVALID_TERM, Last received: 0.0, Next index: 98, Last known committed idx: 97, Time since last communication: 0.000s
I20250112 20:51:41.737128 3812 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Peer 710959530a634cc991544c14efad9e96 log is divergent from this leader: its last log entry 46.98 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 97
I20250112 20:51:41.737908 3769 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 47 FOLLOWER]: Advancing to term 48
W20250112 20:51:41.738973 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.742835 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.743294 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.745504 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.745573 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.746537 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.746769 3835 tablet_replica.cc:1307] Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 34 similar messages]
W20250112 20:51:41.746937 3835 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.747869 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.751688 3487 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.751817 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.753252 3488 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.753568 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.753957 3487 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.755301 3835 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.758916 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.758919 3840 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.760223 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.761723 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.762672 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.763886 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.765792 3328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.767343 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.769152 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.770063 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.771055 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.772971 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.773366 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.773406 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.775835 3835 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.775826 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.783551 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.783594 3835 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.784895 3486 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.784907 3487 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.785017 3488 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.785540 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.786674 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.790534 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.796854 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.797284 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.798202 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.798504 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.798609 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.799856 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.802213 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.803923 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.811108 3328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.812397 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.812795 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.814136 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.814420 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.815255 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.815817 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.819211 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.825685 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.826967 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.829170 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.830847 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.832212 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.833209 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.833223 3837 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.839332 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.841755 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.842468 3328 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.843276 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.845906 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.847652 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.850217 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.851593 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.856683 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:41.857642 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:41.857887 3652 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 50 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.857997 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 50 FOLLOWER]: Advancing to term 51
I20250112 20:51:41.861114 3652 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 51 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.861871 3652 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 51 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
W20250112 20:51:41.865341 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.865490 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.866175 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.866942 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 51 candidate_status { last_received { term: 49 index: 74 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.867180 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Advancing to term 51
I20250112 20:51:41.870414 3372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 51 candidate_status { last_received { term: 49 index: 74 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.870678 3372 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 FOLLOWER]: Advancing to term 51
W20250112 20:51:41.871956 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.872680 3354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.873046 3837 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.873366 3372 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 51.
W20250112 20:51:41.873898 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.874161 3578 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 51 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
W20250112 20:51:41.874220 3836 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:41.874338 3652 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:41.874537 3652 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:41.874576 3731 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 51 FOLLOWER]: Leader election won for term 51
I20250112 20:51:41.874670 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 48 FOLLOWER]: Advancing to term 49
I20250112 20:51:41.874743 3731 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 51 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.875013 3731 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 74, Committed index: 74, Last appended: 49.74, Last appended by leader: 74, Current term: 51, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.876881 3510 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 51 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 51.
I20250112 20:51:41.877343 3652 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 49 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.877983 3652 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 49 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:41.878479 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 49 candidate_status { last_received { term: 46 index: 98 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:41.878685 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 48 FOLLOWER]: Advancing to term 49
I20250112 20:51:41.878705 3212 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 49 to 51, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 51 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:41.880295 3372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 49 candidate_status { last_received { term: 46 index: 98 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:41.880489 3372 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 48 FOLLOWER]: Advancing to term 49
I20250112 20:51:41.883004 3372 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 49 because replica has last-logged OpId of term: 47 index: 102, which is greater than that of the candidate, which has last-logged OpId of term: 46 index: 98.
I20250112 20:51:41.884588 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 49 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 49.
I20250112 20:51:41.885046 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 49 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:41.885569 3824 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 49 FOLLOWER]: Leader election won for term 49
I20250112 20:51:41.885711 3824 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 49 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:41.885998 3824 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 97, Committed index: 97, Last appended: 46.98, Last appended by leader: 98, Current term: 49, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:41.886153 3372 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 51 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 49 index: 74. Preceding OpId from leader: term: 51 index: 76. (index mismatch)
I20250112 20:51:41.886161 3510 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 51 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 49 index: 73. Preceding OpId from leader: term: 51 index: 76. (index mismatch)
I20250112 20:51:41.887005 3824 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 73, Time since last communication: 0.000s
W20250112 20:51:41.887595 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.887668 3824 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 75, Last known committed idx: 74, Time since last communication: 0.000s
W20250112 20:51:41.888617 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.889891 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.901095 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 47 to 49, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 49 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:41.907822 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.909587 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.915338 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.918977 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.922353 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.925901 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:41.929082 3372 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 47 index: 102. Preceding OpId from leader: term: 49 index: 100. (term mismatch)
I20250112 20:51:41.929291 3372 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 99
I20250112 20:51:41.929425 3372 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 47.100
W20250112 20:51:41.929651 3372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Aborted: Op aborted by new leader
I20250112 20:51:41.930006 3372 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 47.101
W20250112 20:51:41.930281 3372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
I20250112 20:51:41.930342 3511 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 49 FOLLOWER]: Refusing update from remote peer 710959530a634cc991544c14efad9e96: Log matching property violated. Preceding OpId in replica: term: 46 index: 97. Preceding OpId from leader: term: 49 index: 100. (index mismatch)
I20250112 20:51:41.930842 3372 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted PARTICIPANT_OP operation due to leader change: 47.102
I20250112 20:51:41.930888 3824 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 92, Time since last communication: 0.000s
W20250112 20:51:41.930999 3372 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Aborted: Op aborted by new leader
I20250112 20:51:41.933933 3722 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 92, Time since last communication: 0.000s
I20250112 20:51:41.934132 3722 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Peer d3f9c3f8fe0d4459b54d92f75a2c84b7 log is divergent from this leader: its last log entry 47.99 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 92
I20250112 20:51:41.935329 3375 raft_consensus.cc:1212] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Deduplicated request from leader. Original: 46.92->[46.93-49.101] Dedup: 46.97->[46.98-49.101]
I20250112 20:51:41.935503 3375 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting all ops after (but not including) 97
I20250112 20:51:41.935611 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted NO_OP operation due to leader change: 47.98
I20250112 20:51:41.935703 3375 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:41.935789 3375 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7: Aborting uncommitted WRITE_OP operation due to leader change: 47.99
W20250112 20:51:41.935961 3375 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Aborted: Op aborted by new leader
W20250112 20:51:41.938231 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.939249 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.941005 3823 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:41.942190 3823 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: Cannot bump the last assigned timestamp. Tablet is not in leader mode. Last heard from a leader: 0.005s ago.
W20250112 20:51:41.954960 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.955412 3465 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.957545 3785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.959388 3786 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.960477 3846 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:41.974884 3839 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:42.019685 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:42.020476 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:42.020967 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 51 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.021153 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 51 FOLLOWER]: Advancing to term 52
I20250112 20:51:42.028743 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 52 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.029810 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 52 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.031308 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 52 candidate_status { last_received { term: 51 index: 94 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.031643 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 51 FOLLOWER]: Advancing to term 52
I20250112 20:51:42.036063 3510 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 52 because replica has last-logged OpId of term: 51 index: 95, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 94.
I20250112 20:51:42.035463 3374 raft_consensus.cc:1235] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 52 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 51. Current term is 52. Ops: [51.95-51.95]
I20250112 20:51:42.040948 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:42.041188 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.041318 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 49 FOLLOWER]: Advancing to term 50
I20250112 20:51:42.042846 3722 consensus_queue.cc:1046] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: INVALID_TERM, Last received: 51.94, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20250112 20:51:42.043224 3722 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 51 LEADER]: Stepping down as leader of term 51
I20250112 20:51:42.043367 3722 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 51 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.043807 3722 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 51.99, Last appended by leader: 99, Current term: 51, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.044304 3722 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 51 FOLLOWER]: Advancing to term 52
I20250112 20:51:42.044981 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.045840 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 50 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.046413 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 50 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.046640 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 49 FOLLOWER]: Advancing to term 50
I20250112 20:51:42.048426 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 50.
I20250112 20:51:42.048910 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 50 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:42.049219 3769 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 FOLLOWER]: Leader election won for term 50
I20250112 20:51:42.049365 3769 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:42.049645 3769 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 49.118, Last appended by leader: 118, Current term: 50, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.049772 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 52 candidate_status { last_received { term: 51 index: 94 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.049856 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 50 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.050019 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 52 because replica has last-logged OpId of term: 51 index: 99, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 94.
I20250112 20:51:42.050040 3651 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 49 LEADER]: Stepping down as leader of term 49
I20250112 20:51:42.050125 3651 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 49 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.050411 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 52 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb
I20250112 20:51:42.050374 3651 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 49.122, Last appended by leader: 122, Current term: 49, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.050580 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 49 FOLLOWER]: Advancing to term 50
I20250112 20:51:42.050650 3769 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 52 FOLLOWER]: Leader election lost for term 52. Reason: could not achieve majority
I20250112 20:51:42.052332 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 49 to 50, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 50 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.052685 3651 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 50 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 50 because replica has last-logged OpId of term: 49 index: 122, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 118.
I20250112 20:51:42.052928 3510 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 49. Current term is 50. Ops: [49.119-49.122]
I20250112 20:51:42.053638 3375 raft_consensus.cc:1235] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 LEADER]: Rejecting Update request from peer 710959530a634cc991544c14efad9e96 for earlier term 49. Current term is 50. Ops: [49.119-49.122]
I20250112 20:51:42.137509 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:42.138484 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:42.138736 3510 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 52 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.138837 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 52 FOLLOWER]: Advancing to term 53
I20250112 20:51:42.140810 3510 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 53 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.141438 3510 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 53 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.142010 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 53 candidate_status { last_received { term: 51 index: 95 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.141995 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 53 candidate_status { last_received { term: 51 index: 95 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.142225 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 52 FOLLOWER]: Advancing to term 53
I20250112 20:51:42.142230 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 52 FOLLOWER]: Advancing to term 53
I20250112 20:51:42.142616 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:42.142812 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.142930 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 50 FOLLOWER]: Advancing to term 51
I20250112 20:51:42.144331 3375 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 53.
I20250112 20:51:42.144819 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:42.145102 3651 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 53 because replica has last-logged OpId of term: 51 index: 99, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 95.
I20250112 20:51:42.145113 3746 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 53 FOLLOWER]: Leader election won for term 53
I20250112 20:51:42.145262 3746 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 53 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:42.145555 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 51 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.145591 3746 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 51.95, Last appended by leader: 95, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.146236 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 51 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.146668 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 51 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.146812 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 51 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.146875 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 50 FOLLOWER]: Advancing to term 51
I20250112 20:51:42.146998 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 LEADER]: Stepping down as leader of term 50
I20250112 20:51:42.147075 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:42.147469 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 50.119, Last appended by leader: 119, Current term: 50, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.147742 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 50 FOLLOWER]: Advancing to term 51
I20250112 20:51:42.148335 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 51 to 53, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 53 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.149103 3651 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 51 because replica has last-logged OpId of term: 49 index: 122, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 118.
I20250112 20:51:42.150410 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 51 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 51 because replica has last-logged OpId of term: 50 index: 119, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 118.
I20250112 20:51:42.150877 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 51 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:42.151225 3746 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 51 FOLLOWER]: Leader election lost for term 51. Reason: could not achieve majority
I20250112 20:51:42.244808 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:42.245743 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:42.245987 3651 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 53 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.246098 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 53 FOLLOWER]: Advancing to term 54
I20250112 20:51:42.247962 3651 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 54 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.248625 3651 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 54 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:42.249135 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 54 candidate_status { last_received { term: 51 index: 99 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.249137 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 54 candidate_status { last_received { term: 51 index: 99 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.249351 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 FOLLOWER]: Advancing to term 54
I20250112 20:51:42.249370 3510 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 53 LEADER]: Stepping down as leader of term 53
I20250112 20:51:42.249447 3510 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 53 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:42.249800 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:42.249769 3510 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 53.96, Last appended by leader: 96, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.250006 3651 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 51 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.250126 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 51 FOLLOWER]: Advancing to term 52
I20250112 20:51:42.250175 3510 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 53 FOLLOWER]: Advancing to term 54
I20250112 20:51:42.252007 3651 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 52 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.252089 3375 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 54 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 54.
I20250112 20:51:42.252705 3651 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 52 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:42.252681 3578 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 54 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:42.252977 3731 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 54 FOLLOWER]: Leader election won for term 54
I20250112 20:51:42.253119 3731 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 54 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.253146 3510 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 54 because replica has last-logged OpId of term: 53 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 99.
I20250112 20:51:42.253408 3731 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 51.99, Last appended by leader: 99, Current term: 54, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.254254 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 52 candidate_status { last_received { term: 49 index: 122 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.254247 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 52 candidate_status { last_received { term: 49 index: 122 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.254426 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 51 FOLLOWER]: Advancing to term 52
I20250112 20:51:42.254436 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 51 FOLLOWER]: Advancing to term 52
I20250112 20:51:42.256218 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 53 to 54, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 54 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:42.256503 3511 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 52 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 52.
I20250112 20:51:42.256682 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 52 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 52 because replica has last-logged OpId of term: 50 index: 119, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 122.
I20250112 20:51:42.256987 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 52 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:42.257253 3731 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 52 FOLLOWER]: Leader election won for term 52
I20250112 20:51:42.257463 3731 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 52 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.257683 3731 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 49.122, Last appended by leader: 122, Current term: 52, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.260999 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 50 to 52, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 52 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:42.395668 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:42.396549 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:42.396802 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 54 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.396921 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 54 FOLLOWER]: Advancing to term 55
I20250112 20:51:42.398859 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 55 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.399550 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 55 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.400030 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 55 candidate_status { last_received { term: 51 index: 94 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.400116 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 55 candidate_status { last_received { term: 51 index: 94 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.400290 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 54 FOLLOWER]: Advancing to term 55
I20250112 20:51:42.400300 3651 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 54 LEADER]: Stepping down as leader of term 54
I20250112 20:51:42.400380 3651 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 54 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.400693 3651 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 54.100, Last appended by leader: 100, Current term: 54, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.400789 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:42.400985 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 52 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.400997 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 54 FOLLOWER]: Advancing to term 55
I20250112 20:51:42.401111 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 52 FOLLOWER]: Advancing to term 53
I20250112 20:51:42.403311 3511 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 55 because replica has last-logged OpId of term: 53 index: 96, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 94.
I20250112 20:51:42.403962 3651 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 55 because replica has last-logged OpId of term: 54 index: 100, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 94.
I20250112 20:51:42.404057 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.404397 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 55 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb
I20250112 20:51:42.404764 3721 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 55 FOLLOWER]: Leader election lost for term 55. Reason: could not achieve majority
I20250112 20:51:42.404855 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 53 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.405359 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 53 candidate_status { last_received { term: 50 index: 119 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.405489 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 53 candidate_status { last_received { term: 50 index: 119 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.405671 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 52 FOLLOWER]: Advancing to term 53
I20250112 20:51:42.405697 3651 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 52 LEADER]: Stepping down as leader of term 52
I20250112 20:51:42.405786 3651 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 52 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.406085 3651 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 52.123, Last appended by leader: 123, Current term: 52, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.406409 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 52 FOLLOWER]: Advancing to term 53
I20250112 20:51:42.407689 3511 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 53 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 53.
I20250112 20:51:42.408124 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 53 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:42.408464 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 FOLLOWER]: Leader election won for term 53
I20250112 20:51:42.408679 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:42.408967 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 50.119, Last appended by leader: 119, Current term: 53, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.410549 3651 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 53 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 53 because replica has last-logged OpId of term: 52 index: 123, which is greater than that of the candidate, which has last-logged OpId of term: 50 index: 119.
I20250112 20:51:42.411782 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 52 to 53, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 53 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.436511 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:42.437631 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:42.437958 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 55 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.438081 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 55 FOLLOWER]: Advancing to term 56
I20250112 20:51:42.440127 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 56 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.440912 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 56 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.441522 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 56 candidate_status { last_received { term: 53 index: 96 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.441522 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 56 candidate_status { last_received { term: 53 index: 96 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.441779 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 55 FOLLOWER]: Advancing to term 56
I20250112 20:51:42.441766 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 55 FOLLOWER]: Advancing to term 56
I20250112 20:51:42.442143 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:42.442329 3511 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 53 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.442477 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 53 FOLLOWER]: Advancing to term 54
I20250112 20:51:42.444025 3375 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 56.
I20250112 20:51:42.444518 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:42.444692 3651 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 56 because replica has last-logged OpId of term: 54 index: 100, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 96.
I20250112 20:51:42.444912 3746 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 56 FOLLOWER]: Leader election won for term 56
I20250112 20:51:42.445116 3746 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 56 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:42.445128 3511 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 54 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.445459 3746 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 53.96, Last appended by leader: 96, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.446230 3511 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 54 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.446393 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 54 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.446592 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 53 FOLLOWER]: Advancing to term 54
I20250112 20:51:42.447522 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 54 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.447775 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 LEADER]: Stepping down as leader of term 53
I20250112 20:51:42.447892 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:42.448173 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 53.120, Last appended by leader: 120, Current term: 53, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.448459 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 53 FOLLOWER]: Advancing to term 54
I20250112 20:51:42.448504 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 54 to 56, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 56 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.448891 3651 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 54 because replica has last-logged OpId of term: 52 index: 123, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 118.
I20250112 20:51:42.450879 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 54 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 54 because replica has last-logged OpId of term: 53 index: 120, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 118.
I20250112 20:51:42.451347 3437 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 54 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:42.451710 3746 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 54 FOLLOWER]: Leader election lost for term 54. Reason: could not achieve majority
I20250112 20:51:42.624814 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 2
I20250112 20:51:42.625808 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:42.626041 3651 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 56 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.626153 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 56 FOLLOWER]: Advancing to term 57
I20250112 20:51:42.628077 3651 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 57 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.628713 3651 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 57 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899)
I20250112 20:51:42.629267 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 57 candidate_status { last_received { term: 54 index: 100 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.629271 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 57 candidate_status { last_received { term: 54 index: 100 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.629491 3511 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 56 LEADER]: Stepping down as leader of term 56
I20250112 20:51:42.629491 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 FOLLOWER]: Advancing to term 57
I20250112 20:51:42.629587 3511 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 56 LEADER]: Becoming Follower/Learner. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:42.629995 3651 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "710959530a634cc991544c14efad9e96"
from {username='slave'} at 127.0.0.1:48046
I20250112 20:51:42.629933 3511 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 56.97, Last appended by leader: 97, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.630196 3651 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 54 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.630226 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 56 FOLLOWER]: Advancing to term 57
I20250112 20:51:42.630326 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 54 FOLLOWER]: Advancing to term 55
I20250112 20:51:42.632225 3375 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 57 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 57.
I20250112 20:51:42.632292 3651 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 55 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.632737 3578 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 57 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:42.632962 3511 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 57 because replica has last-logged OpId of term: 56 index: 97, which is greater than that of the candidate, which has last-logged OpId of term: 54 index: 100.
I20250112 20:51:42.633087 3731 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 57 FOLLOWER]: Leader election won for term 57
I20250112 20:51:42.633113 3651 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 55 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), f15597f529254479b5987d2613686dfb (127.0.104.2:44825)
I20250112 20:51:42.633255 3731 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 57 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.633548 3731 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 54.100, Last appended by leader: 100, Current term: 57, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.633680 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 55 candidate_status { last_received { term: 52 index: 123 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.633869 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 54 FOLLOWER]: Advancing to term 55
I20250112 20:51:42.634219 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "710959530a634cc991544c14efad9e96" candidate_term: 55 candidate_status { last_received { term: 52 index: 123 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.634398 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 54 FOLLOWER]: Advancing to term 55
I20250112 20:51:42.636355 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 55 FOLLOWER]: Leader election vote request: Denying vote to candidate 710959530a634cc991544c14efad9e96 for term 55 because replica has last-logged OpId of term: 53 index: 120, which is greater than that of the candidate, which has last-logged OpId of term: 52 index: 123.
I20250112 20:51:42.636489 3511 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 55 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 710959530a634cc991544c14efad9e96 in term 55.
I20250112 20:51:42.636595 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 56 to 57, leader changed from f15597f529254479b5987d2613686dfb (127.0.104.2) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 57 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:42.636966 3580 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [CANDIDATE]: Term 55 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb; no voters: d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:42.637264 3731 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 55 FOLLOWER]: Leader election won for term 55
I20250112 20:51:42.637425 3731 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 55 LEADER]: Becoming Leader. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.637670 3731 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 52.123, Last appended by leader: 123, Current term: 55, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.641395 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 reported cstate change: term changed from 53 to 55, leader changed from d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1) to 710959530a634cc991544c14efad9e96 (127.0.104.3). New cstate: current_term: 55 leader_uuid: "710959530a634cc991544c14efad9e96" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:42.673960 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 0
I20250112 20:51:42.675175 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:42.675405 3375 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 57 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.675549 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 57 FOLLOWER]: Advancing to term 58
I20250112 20:51:42.677572 3375 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 58 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.678244 3375 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 58 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.678814 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 58 candidate_status { last_received { term: 51 index: 94 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.678838 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 58 candidate_status { last_received { term: 51 index: 94 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.679049 3651 raft_consensus.cc:3049] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 57 LEADER]: Stepping down as leader of term 57
I20250112 20:51:42.679144 3651 raft_consensus.cc:738] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 57 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.679165 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 57 FOLLOWER]: Advancing to term 58
I20250112 20:51:42.679448 3651 consensus_queue.cc:260] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 57.101, Last appended by leader: 101, Current term: 57, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.679643 3375 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
from {username='slave'} at 127.0.0.1:56794
I20250112 20:51:42.679764 3651 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 57 FOLLOWER]: Advancing to term 58
I20250112 20:51:42.679889 3375 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 55 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.680038 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 55 FOLLOWER]: Advancing to term 56
I20250112 20:51:42.681988 3375 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.682360 3511 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 58 because replica has last-logged OpId of term: 56 index: 97, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 94.
I20250112 20:51:42.682713 3375 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 56 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.682498 3651 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 58 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 58 because replica has last-logged OpId of term: 57 index: 101, which is greater than that of the candidate, which has last-logged OpId of term: 51 index: 94.
I20250112 20:51:42.683300 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 56 candidate_status { last_received { term: 53 index: 120 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.683346 3511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 56 candidate_status { last_received { term: 53 index: 120 } } ignore_live_leader: true dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:42.683388 3303 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 58 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters: 710959530a634cc991544c14efad9e96, f15597f529254479b5987d2613686dfb
I20250112 20:51:42.683506 3652 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 55 LEADER]: Stepping down as leader of term 55
I20250112 20:51:42.683598 3511 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 55 FOLLOWER]: Advancing to term 56
I20250112 20:51:42.683605 3652 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 55 LEADER]: Becoming Follower/Learner. State: Replica: 710959530a634cc991544c14efad9e96, State: Running, Role: LEADER
I20250112 20:51:42.683794 3721 raft_consensus.cc:2743] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 58 FOLLOWER]: Leader election lost for term 58. Reason: could not achieve majority
I20250112 20:51:42.683938 3652 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 55.124, Last appended by leader: 124, Current term: 55, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.684190 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 55 FOLLOWER]: Advancing to term 56
I20250112 20:51:42.685743 3511 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 56 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 56.
I20250112 20:51:42.686125 3652 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 56 FOLLOWER]: Leader election vote request: Denying vote to candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 for term 56 because replica has last-logged OpId of term: 55 index: 124, which is greater than that of the candidate, which has last-logged OpId of term: 53 index: 120.
I20250112 20:51:42.686136 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 56 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:42.686410 3721 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 FOLLOWER]: Leader election won for term 56
I20250112 20:51:42.686582 3721 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:42.686812 3721 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 53.120, Last appended by leader: 120, Current term: 56, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.689107 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 55 to 56, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1). New cstate: current_term: 56 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.878456 416 txn_write_ops-itest.cc:638] attempting to promote replicas at tserver 1
I20250112 20:51:42.879514 3511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:42.879750 3511 raft_consensus.cc:491] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 58 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.879887 3511 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 58 FOLLOWER]: Advancing to term 59
I20250112 20:51:42.881974 3511 raft_consensus.cc:513] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 59 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.882648 3511 leader_election.cc:290] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 59 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.883265 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 59 candidate_status { last_received { term: 56 index: 97 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.883275 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8bedc489994a98b60f7742ba6e3f5c" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 59 candidate_status { last_received { term: 56 index: 97 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.883486 3375 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 58 FOLLOWER]: Advancing to term 59
I20250112 20:51:42.883502 3652 raft_consensus.cc:3054] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 58 FOLLOWER]: Advancing to term 59
I20250112 20:51:42.883874 3510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77"
dest_uuid: "f15597f529254479b5987d2613686dfb"
from {username='slave'} at 127.0.0.1:45774
I20250112 20:51:42.884076 3510 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 56 FOLLOWER]: Starting forced leader election (received explicit request)
I20250112 20:51:42.884177 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 56 FOLLOWER]: Advancing to term 57
I20250112 20:51:42.886368 3375 raft_consensus.cc:2463] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 59 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f15597f529254479b5987d2613686dfb in term 59.
I20250112 20:51:42.886445 3652 raft_consensus.cc:2405] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 59 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 59 because replica has last-logged OpId of term: 57 index: 101, which is greater than that of the candidate, which has last-logged OpId of term: 56 index: 97.
I20250112 20:51:42.886931 3437 leader_election.cc:304] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 59 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3f9c3f8fe0d4459b54d92f75a2c84b7, f15597f529254479b5987d2613686dfb; no voters:
I20250112 20:51:42.886991 3510 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 57 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.887253 3746 raft_consensus.cc:2798] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 59 FOLLOWER]: Leader election won for term 59
I20250112 20:51:42.887421 3746 raft_consensus.cc:695] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [term 59 LEADER]: Becoming Leader. State: Replica: f15597f529254479b5987d2613686dfb, State: Running, Role: LEADER
I20250112 20:51:42.887632 3510 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 57 election: Requested vote from peers d3f9c3f8fe0d4459b54d92f75a2c84b7 (127.0.104.1:42899), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:42.887674 3746 consensus_queue.cc:237] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 94, Committed index: 94, Last appended: 56.97, Last appended by leader: 97, Current term: 59, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.888360 3375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 57 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7"
I20250112 20:51:42.888597 3375 raft_consensus.cc:3049] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 LEADER]: Stepping down as leader of term 56
I20250112 20:51:42.888686 3375 raft_consensus.cc:738] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 LEADER]: Becoming Follower/Learner. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:42.889068 3375 consensus_queue.cc:260] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 56.121, Last appended by leader: 121, Current term: 56, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:42.889571 3375 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 56 FOLLOWER]: Advancing to term 57
I20250112 20:51:42.891116 3652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "f15597f529254479b5987d2613686dfb" candidate_term: 57 candidate_status { last_received { term: 49 index: 118 } } ignore_live_leader: true dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:42.891304 3652 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 56 FOLLOWER]: Advancing to term 57
I20250112 20:51:42.891546 3211 catalog_manager.cc:5526] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb reported cstate change: term changed from 57 to 59, leader changed from 710959530a634cc991544c14efad9e96 (127.0.104.3) to f15597f529254479b5987d2613686dfb (127.0.104.2). New cstate: current_term: 59 leader_uuid: "f15597f529254479b5987d2613686dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:42.892443 3375 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 57 because replica has last-logged OpId of term: 56 index: 121, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 118.
I20250112 20:51:42.894071 3652 raft_consensus.cc:2405] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 57 FOLLOWER]: Leader election vote request: Denying vote to candidate f15597f529254479b5987d2613686dfb for term 57 because replica has last-logged OpId of term: 55 index: 124, which is greater than that of the candidate, which has last-logged OpId of term: 49 index: 118.
I20250112 20:51:42.894490 3439 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [CANDIDATE]: Term 57 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f15597f529254479b5987d2613686dfb; no voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7
I20250112 20:51:42.894809 3746 raft_consensus.cc:2743] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 57 FOLLOWER]: Leader election lost for term 57. Reason: could not achieve majority
I20250112 20:51:43.314718 3375 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 59 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 51 index: 94. Preceding OpId from leader: term: 59 index: 98. (index mismatch)
I20250112 20:51:43.315320 3746 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 93, Time since last communication: 0.000s
I20250112 20:51:43.322054 3652 raft_consensus.cc:1270] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 59 FOLLOWER]: Refusing update from remote peer f15597f529254479b5987d2613686dfb: Log matching property violated. Preceding OpId in replica: term: 57 index: 101. Preceding OpId from leader: term: 59 index: 98. (term mismatch)
I20250112 20:51:43.322207 3652 pending_rounds.cc:77] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting all ops after (but not including) 97
I20250112 20:51:43.322275 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 51.98
W20250112 20:51:43.322474 3652 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:43.322721 3652 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=113, attempt_no=0}) took 1288 ms. Trace:
I20250112 20:51:43.322856 3652 rpcz_store.cc:276] 0112 20:51:42.033893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.033945 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:43.322680 (+1288735us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.037348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8c8bedc489994a98b60f7742ba6e3f5c
0112 20:51:42.037437 (+ 89us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.037440 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:51:42.037452 (+ 12us) tablet.cc:659] Decoding operations
0112 20:51:42.037574 (+ 122us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.037615 (+ 41us) tablet.cc:698] Row locks acquired
0112 20:51:42.037617 (+ 2us) write_op.cc:260] PREPARE: finished
0112 20:51:42.037656 (+ 39us) write_op.cc:270] Start()
0112 20:51:42.037680 (+ 24us) write_op.cc:276] Timestamp: P: 1736715102037596 usec, L: 0
0112 20:51:42.037683 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.037930 (+ 247us) log.cc:844] Serialized 302 byte log entry
0112 20:51:43.322389 (+1284459us) op_driver.cc:464] REPLICATION: finished
0112 20:51:43.322399 (+ 10us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:43.322406 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:43.322420 (+ 14us) write_op.cc:454] Released schema lock
0112 20:51:43.322426 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":2868,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":1397,"replication_time_us":1284638}]]}
I20250112 20:51:43.324334 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 51.99
W20250112 20:51:43.324529 3652 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:43.324779 3652 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=114, attempt_no=0}) took 1285 ms. Trace:
I20250112 20:51:43.324894 3652 rpcz_store.cc:276] 0112 20:51:42.038857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.039037 (+ 180us) service_pool.cc:225] Handling call
0112 20:51:43.324731 (+1285694us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.039981 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8c8bedc489994a98b60f7742ba6e3f5c
0112 20:51:42.040080 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.040084 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:51:42.040086 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:42.040151 (+ 65us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.040191 (+ 40us) tablet.cc:698] Row locks acquired
0112 20:51:42.040193 (+ 2us) write_op.cc:260] PREPARE: finished
0112 20:51:42.040229 (+ 36us) write_op.cc:270] Start()
0112 20:51:42.040254 (+ 25us) write_op.cc:276] Timestamp: P: 1736715102040170 usec, L: 0
0112 20:51:42.040258 (+ 4us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.040406 (+ 148us) log.cc:844] Serialized 302 byte log entry
0112 20:51:43.324440 (+1284034us) op_driver.cc:464] REPLICATION: finished
0112 20:51:43.324449 (+ 9us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:43.324455 (+ 6us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:43.324467 (+ 12us) write_op.cc:454] Released schema lock
0112 20:51:43.324472 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":469,"prepare.run_cpu_time_us":492,"prepare.run_wall_time_us":825,"replication_time_us":1284136}]]}
I20250112 20:51:43.325313 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted NO_OP operation due to leader change: 54.100
I20250112 20:51:43.325409 3652 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 59 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:43.325498 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted NO_OP operation due to leader change: 57.101
I20250112 20:51:43.325577 3652 raft_consensus.cc:2883] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96 [term 59 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:43.326493 3856 consensus_queue.cc:1035] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 95, Time since last communication: 0.000s
I20250112 20:51:43.326714 3856 consensus_queue.cc:1225] T 8c8bedc489994a98b60f7742ba6e3f5c P f15597f529254479b5987d2613686dfb [LEADER]: Peer 710959530a634cc991544c14efad9e96 log is divergent from this leader: its last log entry 51.97 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 95
I20250112 20:51:43.327621 3652 pending_rounds.cc:77] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting all ops after (but not including) 95
I20250112 20:51:43.327755 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 51.96
W20250112 20:51:43.327890 3652 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:43.328091 3652 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=109, attempt_no=0}) took 1301 ms. Trace:
I20250112 20:51:43.328203 3652 rpcz_store.cc:276] 0112 20:51:42.027071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.029198 (+ 2127us) service_pool.cc:225] Handling call
0112 20:51:43.328038 (+1298840us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.029980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8c8bedc489994a98b60f7742ba6e3f5c
0112 20:51:42.030092 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.030096 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:51:42.030098 (+ 2us) tablet.cc:659] Decoding operations
0112 20:51:42.030178 (+ 80us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.030229 (+ 51us) tablet.cc:698] Row locks acquired
0112 20:51:42.030233 (+ 4us) write_op.cc:260] PREPARE: finished
0112 20:51:42.030288 (+ 55us) write_op.cc:270] Start()
0112 20:51:42.030322 (+ 34us) write_op.cc:276] Timestamp: P: 1736715102030224 usec, L: 0
0112 20:51:42.030328 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.030556 (+ 228us) log.cc:844] Serialized 302 byte log entry
0112 20:51:43.327838 (+1297282us) op_driver.cc:464] REPLICATION: finished
0112 20:51:43.327848 (+ 10us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:43.327852 (+ 4us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:43.327863 (+ 11us) write_op.cc:454] Released schema lock
0112 20:51:43.327867 (+ 4us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":6387,"replication_time_us":1297465}]]}
I20250112 20:51:43.329018 3652 pending_rounds.cc:99] T 8c8bedc489994a98b60f7742ba6e3f5c P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 51.97
W20250112 20:51:43.329192 3652 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:43.329408 3652 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=110, attempt_no=0}) took 1299 ms. Trace:
I20250112 20:51:43.329505 3652 rpcz_store.cc:276] 0112 20:51:42.030114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.030166 (+ 52us) service_pool.cc:225] Handling call
0112 20:51:43.329349 (+1299183us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.036408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 8c8bedc489994a98b60f7742ba6e3f5c
0112 20:51:42.036530 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.036536 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:42.036540 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:42.036651 (+ 111us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.036701 (+ 50us) tablet.cc:698] Row locks acquired
0112 20:51:42.036703 (+ 2us) write_op.cc:260] PREPARE: finished
0112 20:51:42.036743 (+ 40us) write_op.cc:270] Start()
0112 20:51:42.036773 (+ 30us) write_op.cc:276] Timestamp: P: 1736715102036682 usec, L: 0
0112 20:51:42.036776 (+ 3us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.036950 (+ 174us) log.cc:844] Serialized 302 byte log entry
0112 20:51:43.329125 (+1292175us) op_driver.cc:464] REPLICATION: finished
0112 20:51:43.329132 (+ 7us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:43.329137 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:43.329148 (+ 11us) write_op.cc:454] Released schema lock
0112 20:51:43.329153 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":5762,"prepare.run_cpu_time_us":614,"prepare.run_wall_time_us":909,"replication_time_us":1292311}]]}
W20250112 20:51:43.360028 3629 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.360710 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.368278 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.369928 3855 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations [suppressed 20 similar messages]
W20250112 20:51:43.373057 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.376495 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.380249 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.383052 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.383697 3864 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations [suppressed 13 similar messages]
W20250112 20:51:43.387590 3875 tablet_replica.cc:1307] Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 14 similar messages]
W20250112 20:51:43.388095 3875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.389504 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.390401 3875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.391182 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.395922 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.398245 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.398291 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.398566 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.402580 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.405596 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.406988 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.408129 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.410408 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.412808 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.416036 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.419725 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.420254 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.420504 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.424842 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.429699 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.431283 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.435104 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.439771 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.440580 3875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.444842 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.445430 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.449627 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.450700 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.457243 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.458389 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.460810 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.461921 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.471755 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.475540 3875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.475540 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.477244 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.489946 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.490420 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.492350 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.492600 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.504441 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.505333 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.505864 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.506996 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.522069 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.524160 3875 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.524304 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.524363 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.543241 3861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.544171 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.544346 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.545094 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.559670 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.560936 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.561031 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.563477 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.581846 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.583812 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.583892 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.584870 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.602164 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.606359 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.607375 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.607443 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.621490 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.627434 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.630118 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.631036 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.648880 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.654628 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.654614 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.658228 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.674652 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.678571 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.680529 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.684711 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.698540 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.704329 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.705150 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.707069 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.729008 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.730300 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.732757 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.734375 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.756021 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.757704 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.763098 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.763800 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.783142 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.784176 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.790983 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.793172 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.813149 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.813246 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.821728 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.824779 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.843282 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.843326 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.855460 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.856000 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.872232 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.873270 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.885345 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.889153 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.905993 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.906850 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.920737 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.921314 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.939397 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.940210 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.955413 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:43.958509 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.972451 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.973503 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.991245 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:43.993950 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.011790 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.012639 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.028714 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.030490 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.049678 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.051393 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.067302 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.069264 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.086443 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.087489 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.102241 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.108065 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.126905 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.126922 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.141584 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.145750 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.165938 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.166970 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.182384 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.184677 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.205660 3602 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.205660 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.223894 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.227281 3489 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.249017 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.250528 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.266583 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.269714 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.291997 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.292044 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.307392 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.315376 3353 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:35201: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.334399 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.335484 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.347286 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.360502 3491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:50099: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.381714 3876 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.381747 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.393090 3877 tablet_replica.cc:1307] Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 40 similar messages]
W20250112 20:51:44.393260 3877 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56804: Illegal state: replica d3f9c3f8fe0d4459b54d92f75a2c84b7 is not leader of this config: current role FOLLOWER
I20250112 20:51:44.402285 3878 raft_consensus.cc:491] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 57 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:44.402518 3878 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 57 FOLLOWER]: Advancing to term 58
I20250112 20:51:44.405062 3878 raft_consensus.cc:513] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 58 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:44.405772 3878 leader_election.cc:290] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 58 election: Requested vote from peers f15597f529254479b5987d2613686dfb (127.0.104.2:44825), 710959530a634cc991544c14efad9e96 (127.0.104.3:44097)
I20250112 20:51:44.406322 3510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 58 candidate_status { last_received { term: 56 index: 121 } } ignore_live_leader: false dest_uuid: "f15597f529254479b5987d2613686dfb"
I20250112 20:51:44.406384 3651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8edbeb2c8364a3691b8d3c1289b1e77" candidate_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" candidate_term: 58 candidate_status { last_received { term: 56 index: 121 } } ignore_live_leader: false dest_uuid: "710959530a634cc991544c14efad9e96"
I20250112 20:51:44.406535 3510 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 57 FOLLOWER]: Advancing to term 58
I20250112 20:51:44.406679 3651 raft_consensus.cc:3054] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 57 FOLLOWER]: Advancing to term 58
W20250112 20:51:44.407387 3630 tablet_service.cc:695] failed op from {username='slave'} at 127.0.104.2:36899: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:44.409085 3651 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 58.
I20250112 20:51:44.409279 3510 raft_consensus.cc:2463] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 58 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f9c3f8fe0d4459b54d92f75a2c84b7 in term 58.
I20250112 20:51:44.409672 3303 leader_election.cc:304] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [CANDIDATE]: Term 58 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 710959530a634cc991544c14efad9e96, d3f9c3f8fe0d4459b54d92f75a2c84b7; no voters:
I20250112 20:51:44.410102 3878 raft_consensus.cc:2798] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 58 FOLLOWER]: Leader election won for term 58
I20250112 20:51:44.410521 3878 raft_consensus.cc:695] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [term 58 LEADER]: Becoming Leader. State: Replica: d3f9c3f8fe0d4459b54d92f75a2c84b7, State: Running, Role: LEADER
I20250112 20:51:44.410845 3878 consensus_queue.cc:237] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 56.121, Last appended by leader: 121, Current term: 58, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } }
I20250112 20:51:44.413594 3211 catalog_manager.cc:5526] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 reported cstate change: term changed from 56 to 58. New cstate: current_term: 58 leader_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f9c3f8fe0d4459b54d92f75a2c84b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:44.427443 3860 tablet_replica.cc:1307] Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER: cancelling pending write operations [suppressed 39 similar messages]
W20250112 20:51:44.427443 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.427634 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.439440 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:44.456332 3510 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P f15597f529254479b5987d2613686dfb [term 58 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 49 index: 118. Preceding OpId from leader: term: 58 index: 123. (index mismatch)
I20250112 20:51:44.456462 3651 raft_consensus.cc:1270] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 58 FOLLOWER]: Refusing update from remote peer d3f9c3f8fe0d4459b54d92f75a2c84b7: Log matching property violated. Preceding OpId in replica: term: 55 index: 124. Preceding OpId from leader: term: 58 index: 123. (term mismatch)
I20250112 20:51:44.456630 3651 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting all ops after (but not including) 122
I20250112 20:51:44.456717 3651 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting uncommitted NO_OP operation due to leader change: 52.123
I20250112 20:51:44.456779 3651 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:44.456856 3651 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting uncommitted NO_OP operation due to leader change: 55.124
I20250112 20:51:44.456908 3651 raft_consensus.cc:2883] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96 [term 58 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250112 20:51:44.457005 3879 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f15597f529254479b5987d2613686dfb" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 116, Time since last communication: 0.000s
I20250112 20:51:44.457508 3879 consensus_queue.cc:1035] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "710959530a634cc991544c14efad9e96" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 118, Time since last communication: 0.000s
I20250112 20:51:44.457653 3879 consensus_queue.cc:1225] T b8edbeb2c8364a3691b8d3c1289b1e77 P d3f9c3f8fe0d4459b54d92f75a2c84b7 [LEADER]: Peer 710959530a634cc991544c14efad9e96 log is divergent from this leader: its last log entry 49.122 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 118
I20250112 20:51:44.458712 3651 pending_rounds.cc:77] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting all ops after (but not including) 118
I20250112 20:51:44.458807 3651 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 49.119
W20250112 20:51:44.459046 3651 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:44.459283 3651 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=112, attempt_no=0}) took 2428 ms. Trace:
I20250112 20:51:44.459394 3651 rpcz_store.cc:276] 0112 20:51:42.030825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.030884 (+ 59us) service_pool.cc:225] Handling call
0112 20:51:44.459251 (+2428367us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.031408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b8edbeb2c8364a3691b8d3c1289b1e77
0112 20:51:42.031539 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.031545 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:42.031548 (+ 3us) tablet.cc:659] Decoding operations
0112 20:51:42.031638 (+ 90us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.031708 (+ 70us) tablet.cc:698] Row locks acquired
0112 20:51:42.031712 (+ 4us) write_op.cc:260] PREPARE: finished
0112 20:51:42.031753 (+ 41us) write_op.cc:270] Start()
0112 20:51:42.031785 (+ 32us) write_op.cc:276] Timestamp: P: 1736715102031692 usec, L: 0
0112 20:51:42.031791 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.031979 (+ 188us) log.cc:844] Serialized 302 byte log entry
0112 20:51:44.458950 (+2426971us) op_driver.cc:464] REPLICATION: finished
0112 20:51:44.458961 (+ 11us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:44.458972 (+ 11us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:44.458994 (+ 22us) write_op.cc:454] Released schema lock
0112 20:51:44.459002 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":647,"prepare.run_wall_time_us":648,"replication_time_us":2427100}]]}
I20250112 20:51:44.459988 3651 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 49.120
W20250112 20:51:44.460168 3651 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:44.460350 3651 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=111, attempt_no=0}) took 2428 ms. Trace:
I20250112 20:51:44.460459 3651 rpcz_store.cc:276] 0112 20:51:42.032196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.032239 (+ 43us) service_pool.cc:225] Handling call
0112 20:51:44.460328 (+2428089us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.032929 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b8edbeb2c8364a3691b8d3c1289b1e77
0112 20:51:42.033077 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.033083 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:42.033088 (+ 5us) tablet.cc:659] Decoding operations
0112 20:51:42.033187 (+ 99us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.033251 (+ 64us) tablet.cc:698] Row locks acquired
0112 20:51:42.033256 (+ 5us) write_op.cc:260] PREPARE: finished
0112 20:51:42.033307 (+ 51us) write_op.cc:270] Start()
0112 20:51:42.033344 (+ 37us) write_op.cc:276] Timestamp: P: 1736715102033243 usec, L: 0
0112 20:51:42.033351 (+ 7us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.033561 (+ 210us) log.cc:844] Serialized 302 byte log entry
0112 20:51:44.460094 (+2426533us) op_driver.cc:464] REPLICATION: finished
0112 20:51:44.460102 (+ 8us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:44.460107 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:44.460119 (+ 12us) write_op.cc:454] Released schema lock
0112 20:51:44.460124 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":151,"prepare.run_cpu_time_us":723,"prepare.run_wall_time_us":724,"replication_time_us":2426700}]]}
I20250112 20:51:44.461431 3651 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 49.121
W20250112 20:51:44.461611 3651 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:44.461783 3651 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=115, attempt_no=0}) took 2422 ms. Trace:
I20250112 20:51:44.461874 3651 rpcz_store.cc:276] 0112 20:51:42.039659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.039720 (+ 61us) service_pool.cc:225] Handling call
0112 20:51:44.461756 (+2422036us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.040491 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b8edbeb2c8364a3691b8d3c1289b1e77
0112 20:51:42.040614 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.040620 (+ 6us) write_op.cc:435] Acquired schema lock
0112 20:51:42.040624 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:42.040706 (+ 82us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.040770 (+ 64us) tablet.cc:698] Row locks acquired
0112 20:51:42.040774 (+ 4us) write_op.cc:260] PREPARE: finished
0112 20:51:42.040816 (+ 42us) write_op.cc:270] Start()
0112 20:51:42.040848 (+ 32us) write_op.cc:276] Timestamp: P: 1736715102040754 usec, L: 0
0112 20:51:42.040853 (+ 5us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.041028 (+ 175us) log.cc:844] Serialized 302 byte log entry
0112 20:51:44.461536 (+2420508us) op_driver.cc:464] REPLICATION: finished
0112 20:51:44.461545 (+ 9us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:44.461552 (+ 7us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:44.461562 (+ 10us) write_op.cc:454] Released schema lock
0112 20:51:44.461567 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":290,"prepare.run_cpu_time_us":606,"prepare.run_wall_time_us":607,"replication_time_us":2420634}]]}
I20250112 20:51:44.462594 3651 pending_rounds.cc:99] T b8edbeb2c8364a3691b8d3c1289b1e77 P 710959530a634cc991544c14efad9e96: Aborting uncommitted WRITE_OP operation due to leader change: 49.122
W20250112 20:51:44.462755 3651 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Aborted: Op aborted by new leader
I20250112 20:51:44.462924 3651 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48056 (ReqId={client: 6509eaf5328c4f288bf121d5e07d6cfe, seq_no=116, attempt_no=0}) took 2421 ms. Trace:
I20250112 20:51:44.463016 3651 rpcz_store.cc:276] 0112 20:51:42.041280 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:42.041320 (+ 40us) service_pool.cc:225] Handling call
0112 20:51:44.462896 (+2421576us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:51:42.041893 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b8edbeb2c8364a3691b8d3c1289b1e77
0112 20:51:42.041995 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:51:42.042000 (+ 5us) write_op.cc:435] Acquired schema lock
0112 20:51:42.042004 (+ 4us) tablet.cc:659] Decoding operations
0112 20:51:42.042102 (+ 98us) tablet.cc:682] Acquiring locks for 1 operations
0112 20:51:42.042145 (+ 43us) tablet.cc:698] Row locks acquired
0112 20:51:42.042149 (+ 4us) write_op.cc:260] PREPARE: finished
0112 20:51:42.042190 (+ 41us) write_op.cc:270] Start()
0112 20:51:42.042221 (+ 31us) write_op.cc:276] Timestamp: P: 1736715102042128 usec, L: 0
0112 20:51:42.042227 (+ 6us) op_driver.cc:348] REPLICATION: starting
0112 20:51:42.042418 (+ 191us) log.cc:844] Serialized 302 byte log entry
0112 20:51:44.462684 (+2420266us) op_driver.cc:464] REPLICATION: finished
0112 20:51:44.462705 (+ 21us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0112 20:51:44.462710 (+ 5us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:51:44.462718 (+ 8us) write_op.cc:454] Released schema lock
0112 20:51:44.462721 (+ 3us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":593,"prepare.run_wall_time_us":592,"replication_time_us":2420419}]]}
W20250112 20:51:44.464771 3891 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
W20250112 20:51:44.465912 3465 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.471019 3860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.471144 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.471819 3859 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
W20250112 20:51:44.473404 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
W20250112 20:51:44.474555 3893 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45798: Illegal state: replica f15597f529254479b5987d2613686dfb is not leader of this config: current role FOLLOWER
I20250112 20:51:44.478607 3885 mvcc.cc:204] Tried to move back new op lower bound from 7113585067848171520 to 7113585067668893696. Current Snapshot: MvccSnapshot[applied={T|T < 7113585067848171520}]
W20250112 20:51:44.487473 3603 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48056: Illegal state: replica 710959530a634cc991544c14efad9e96 is not leader of this config: current role FOLLOWER
I20250112 20:51:44.494585 3888 mvcc.cc:204] Tried to move back new op lower bound from 7113585067848171520 to 7113585060607111168. Current Snapshot: MvccSnapshot[applied={T|T < 7113585067848171520}]
I20250112 20:51:44.689105 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 3286
I20250112 20:51:44.690285 3415 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:44.831269 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 3286
I20250112 20:51:44.867077 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 3422
W20250112 20:51:44.867074 3199 connection.cc:537] server connection from 127.0.104.1:38633 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 20:51:44.868227 3551 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:45.044425 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 3422
W20250112 20:51:45.072254 3580 connection.cc:537] server connection from 127.0.104.2:34941 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250112 20:51:45.072685 3196 connection.cc:537] server connection from 127.0.104.2:55975 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 20:51:45.072957 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 3558
I20250112 20:51:45.073937 3692 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:45.203215 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 3558
W20250112 20:51:45.224638 3197 connection.cc:537] server connection from 127.0.104.3:55091 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250112 20:51:45.224763 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 3181
I20250112 20:51:45.225723 3252 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:45.341084 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 3181
2025-01-12T20:51:45Z chronyd exiting
[ OK ] TxnWriteOpsITest.FrequentElections (7994 ms)
[ RUN ] TxnWriteOpsITest.WriteOpPerf
2025-01-12T20:51:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:45Z Disabled control of system clock
I20250112 20:51:45.410346 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33533
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33533
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false with env {}
W20250112 20:51:45.739048 3932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:45.739333 3932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:45.739411 3932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:45.746740 3932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:45.746866 3932 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:45.746902 3932 flags.cc:425] Enabled experimental flag: --txn_manager_lazily_initialized=false
W20250112 20:51:45.746953 3932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:45.747000 3932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:45.747052 3932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:45.756160 3932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33533
--txn_manager_enabled=true
--txn_manager_lazily_initialized=false
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:45.757848 3932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:45.759510 3932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:45.769731 3938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:45.769764 3937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:45.769692 3940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:45.770094 3932 server_base.cc:1034] running on GCE node
I20250112 20:51:45.771119 3932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:45.772611 3932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:45.773841 3932 hybrid_clock.cc:648] HybridClock initialized: now 1736715105773786 us; error 44 us; skew 500 ppm
I20250112 20:51:45.774183 3932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:45.776507 3932 webserver.cc:458] Webserver started at http://127.0.104.62:38893/ using document root <none> and password file <none>
I20250112 20:51:45.777269 3932 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:45.777406 3932 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:45.777710 3932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:45.780356 3932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "dce89bd2183a49f3ac44d066e45167e9"
format_stamp: "Formatted at 2025-01-12 20:51:45 on dist-test-slave-3gql"
I20250112 20:51:45.781244 3932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "dce89bd2183a49f3ac44d066e45167e9"
format_stamp: "Formatted at 2025-01-12 20:51:45 on dist-test-slave-3gql"
I20250112 20:51:45.786581 3932 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250112 20:51:45.790138 3946 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:45.791463 3932 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250112 20:51:45.791646 3932 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "dce89bd2183a49f3ac44d066e45167e9"
format_stamp: "Formatted at 2025-01-12 20:51:45 on dist-test-slave-3gql"
I20250112 20:51:45.791828 3932 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:45.825065 3932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:45.825958 3932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:45.826246 3932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:45.847259 3932 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33533
I20250112 20:51:45.847316 4007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33533 every 8 connection(s)
I20250112 20:51:45.848768 3932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:45.849473 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 3932
I20250112 20:51:45.849694 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:45.853725 4008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:45.864625 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9: Bootstrap starting.
I20250112 20:51:45.867719 4008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:45.868822 4008 log.cc:826] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:45.871565 4008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9: No bootstrap required, opened a new log
I20250112 20:51:45.878180 4008 raft_consensus.cc:357] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce89bd2183a49f3ac44d066e45167e9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33533 } }
I20250112 20:51:45.878559 4008 raft_consensus.cc:383] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:45.878661 4008 raft_consensus.cc:738] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce89bd2183a49f3ac44d066e45167e9, State: Initialized, Role: FOLLOWER
I20250112 20:51:45.879285 4008 consensus_queue.cc:260] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce89bd2183a49f3ac44d066e45167e9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33533 } }
I20250112 20:51:45.879508 4008 raft_consensus.cc:397] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:45.879617 4008 raft_consensus.cc:491] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:45.879762 4008 raft_consensus.cc:3054] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:45.882151 4008 raft_consensus.cc:513] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce89bd2183a49f3ac44d066e45167e9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33533 } }
I20250112 20:51:45.882789 4008 leader_election.cc:304] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dce89bd2183a49f3ac44d066e45167e9; no voters:
I20250112 20:51:45.883369 4008 leader_election.cc:290] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:45.883545 4013 raft_consensus.cc:2798] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:45.884234 4013 raft_consensus.cc:695] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [term 1 LEADER]: Becoming Leader. State: Replica: dce89bd2183a49f3ac44d066e45167e9, State: Running, Role: LEADER
I20250112 20:51:45.884829 4013 consensus_queue.cc:237] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce89bd2183a49f3ac44d066e45167e9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33533 } }
I20250112 20:51:45.885496 4008 sys_catalog.cc:564] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:45.888046 4015 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dce89bd2183a49f3ac44d066e45167e9. Latest consensus state: current_term: 1 leader_uuid: "dce89bd2183a49f3ac44d066e45167e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce89bd2183a49f3ac44d066e45167e9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33533 } } }
I20250112 20:51:45.888461 4015 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:45.888625 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dce89bd2183a49f3ac44d066e45167e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce89bd2183a49f3ac44d066e45167e9" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33533 } } }
I20250112 20:51:45.888911 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:45.890887 4022 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:45.896138 4022 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:45.911305 4022 catalog_manager.cc:1348] Generated new cluster ID: 0eab0e3fe1204522ae18384ce9aefb87
I20250112 20:51:45.911473 4022 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:45.921552 4022 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:45.922833 4022 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:45.931242 4022 catalog_manager.cc:5899] T 00000000000000000000000000000000 P dce89bd2183a49f3ac44d066e45167e9: Generated new TSK 0
I20250112 20:51:45.932325 4022 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:45.945101 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33533
--builtin_ntp_servers=127.0.104.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tablet_max_pending_txn_write_ops=10
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20250112 20:51:45.941248 3963 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44596:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:45.949832 4008 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 3; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
W20250112 20:51:46.280062 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:46.280362 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:46.280440 4037 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20250112 20:51:46.280529 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:46.288439 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:46.288681 4037 flags.cc:425] Enabled experimental flag: --tablet_max_pending_txn_write_ops=10
W20250112 20:51:46.288796 4037 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:46.288913 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:46.298467 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_partition_lock=false
--tablet_max_pending_txn_write_ops=10
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:33533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:46.300092 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:46.301846 4037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:46.313361 4042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:46.313372 4043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:46.313325 4045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:46.314251 4037 server_base.cc:1034] running on GCE node
I20250112 20:51:46.314956 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:46.316035 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:46.317256 4037 hybrid_clock.cc:648] HybridClock initialized: now 1736715106317206 us; error 41 us; skew 500 ppm
I20250112 20:51:46.317595 4037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:46.319883 4037 webserver.cc:458] Webserver started at http://127.0.104.1:33257/ using document root <none> and password file <none>
I20250112 20:51:46.320693 4037 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:46.320822 4037 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:46.321134 4037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:46.323822 4037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "4144e9bc5a0844bd82425be2d44e32ef"
format_stamp: "Formatted at 2025-01-12 20:51:46 on dist-test-slave-3gql"
I20250112 20:51:46.324630 4037 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "4144e9bc5a0844bd82425be2d44e32ef"
format_stamp: "Formatted at 2025-01-12 20:51:46 on dist-test-slave-3gql"
I20250112 20:51:46.329887 4037 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250112 20:51:46.333355 4051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:46.334762 4037 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250112 20:51:46.334942 4037 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "4144e9bc5a0844bd82425be2d44e32ef"
format_stamp: "Formatted at 2025-01-12 20:51:46 on dist-test-slave-3gql"
I20250112 20:51:46.335179 4037 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:46.350718 4037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:46.351594 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:46.351881 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:46.354753 4037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:46.354875 4037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:46.354996 4037 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:46.355067 4037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:46.394847 4037 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39813
I20250112 20:51:46.394884 4170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39813 every 8 connection(s)
I20250112 20:51:46.396078 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:46.396505 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 4037
I20250112 20:51:46.396824 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:46.398386 4171 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33533
I20250112 20:51:46.398600 4171 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:46.399492 4171 heartbeater.cc:510] Master 127.0.104.62:33533 requested a full tablet report, sending...
I20250112 20:51:46.400198 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33533
--builtin_ntp_servers=127.0.104.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tablet_max_pending_txn_write_ops=10
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20250112 20:51:46.401564 3963 ts_manager.cc:194] Registered new tserver with Master: 4144e9bc5a0844bd82425be2d44e32ef (127.0.104.1:39813)
I20250112 20:51:46.403704 3963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:58419
W20250112 20:51:46.732385 4173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:46.732734 4173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:46.732884 4173 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20250112 20:51:46.732969 4173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:46.740824 4173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:46.741019 4173 flags.cc:425] Enabled experimental flag: --tablet_max_pending_txn_write_ops=10
W20250112 20:51:46.741091 4173 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:46.741207 4173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:46.750370 4173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_partition_lock=false
--tablet_max_pending_txn_write_ops=10
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:33533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:46.752002 4173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:46.753723 4173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:46.764691 4181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:46.764750 4179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:46.764731 4178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:46.765806 4173 server_base.cc:1034] running on GCE node
I20250112 20:51:46.766433 4173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:46.767477 4173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:46.768633 4173 hybrid_clock.cc:648] HybridClock initialized: now 1736715106768588 us; error 44 us; skew 500 ppm
I20250112 20:51:46.768946 4173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:46.771157 4173 webserver.cc:458] Webserver started at http://127.0.104.2:42781/ using document root <none> and password file <none>
I20250112 20:51:46.771952 4173 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:46.772042 4173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:46.772300 4173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:46.776166 4173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "247cc258515140bcb8d780b06c1f608e"
format_stamp: "Formatted at 2025-01-12 20:51:46 on dist-test-slave-3gql"
I20250112 20:51:46.776928 4173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "247cc258515140bcb8d780b06c1f608e"
format_stamp: "Formatted at 2025-01-12 20:51:46 on dist-test-slave-3gql"
I20250112 20:51:46.781915 4173 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20250112 20:51:46.785177 4187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:46.786679 4173 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250112 20:51:46.786825 4173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "247cc258515140bcb8d780b06c1f608e"
format_stamp: "Formatted at 2025-01-12 20:51:46 on dist-test-slave-3gql"
I20250112 20:51:46.786998 4173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:46.808347 4173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:46.809228 4173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:46.809456 4173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:46.812108 4173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:46.812212 4173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:46.812353 4173 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:46.812431 4173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:46.857813 4173 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41359
I20250112 20:51:46.857849 4306 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41359 every 8 connection(s)
I20250112 20:51:46.859067 4173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:46.861124 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 4173
I20250112 20:51:46.861270 4307 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33533
I20250112 20:51:46.861310 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:46.861465 4307 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:46.862160 4307 heartbeater.cc:510] Master 127.0.104.62:33533 requested a full tablet report, sending...
I20250112 20:51:46.863552 3963 ts_manager.cc:194] Registered new tserver with Master: 247cc258515140bcb8d780b06c1f608e (127.0.104.2:41359)
I20250112 20:51:46.864288 3963 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:58949
I20250112 20:51:46.865669 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33533
--builtin_ntp_servers=127.0.104.20:36549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tablet_max_pending_txn_write_ops=10
--enable_txn_partition_lock=false
--enable_txn_system_client_init=true with env {}
I20250112 20:51:46.963441 3963 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44602:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:47.191465 4309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:47.191800 4309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:47.191921 4309 flags.cc:425] Enabled unsafe flag: --enable_txn_partition_lock=false
W20250112 20:51:47.192019 4309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:47.199605 4309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:47.199782 4309 flags.cc:425] Enabled experimental flag: --tablet_max_pending_txn_write_ops=10
W20250112 20:51:47.199868 4309 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:47.199973 4309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:47.209195 4309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_partition_lock=false
--tablet_max_pending_txn_write_ops=10
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:33533
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:47.210704 4309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:47.212319 4309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:47.225628 4320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:47.226346 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:47.226671 4309 server_base.cc:1034] running on GCE node
W20250112 20:51:47.226895 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:47.227692 4309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:47.228998 4309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:47.230254 4309 hybrid_clock.cc:648] HybridClock initialized: now 1736715107230197 us; error 46 us; skew 500 ppm
I20250112 20:51:47.230683 4309 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:47.233641 4309 webserver.cc:458] Webserver started at http://127.0.104.3:37031/ using document root <none> and password file <none>
I20250112 20:51:47.234647 4309 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:47.234779 4309 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:47.235152 4309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:47.238941 4309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "75305c00378d416ea46c1a8b59d7337a"
format_stamp: "Formatted at 2025-01-12 20:51:47 on dist-test-slave-3gql"
I20250112 20:51:47.239998 4309 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "75305c00378d416ea46c1a8b59d7337a"
format_stamp: "Formatted at 2025-01-12 20:51:47 on dist-test-slave-3gql"
I20250112 20:51:47.247083 4309 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20250112 20:51:47.251827 4328 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:47.253598 4309 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:47.253805 4309 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "75305c00378d416ea46c1a8b59d7337a"
format_stamp: "Formatted at 2025-01-12 20:51:47 on dist-test-slave-3gql"
I20250112 20:51:47.254077 4309 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:47.291009 4309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:47.292347 4309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:47.292748 4309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:47.296756 4309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:47.296919 4309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:47.297053 4309 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:47.297120 4309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:47.341902 4309 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:43451
I20250112 20:51:47.341935 4447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:43451 every 8 connection(s)
I20250112 20:51:47.343119 4309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:47.344985 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 4309
I20250112 20:51:47.345184 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpPerf.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:47.345227 4448 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33533
I20250112 20:51:47.345453 4448 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:47.346248 4448 heartbeater.cc:510] Master 127.0.104.62:33533 requested a full tablet report, sending...
I20250112 20:51:47.347846 3962 ts_manager.cc:194] Registered new tserver with Master: 75305c00378d416ea46c1a8b59d7337a (127.0.104.3:43451)
I20250112 20:51:47.348875 3962 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:52935
I20250112 20:51:47.360683 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:47.375424 3962 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44610:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:47.376248 3962 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.403635 4240 tablet_service.cc:1467] Processing CreateTablet for tablet e530036caf1c4eb2ad2356427dd66caf (DEFAULT_TABLE table=txn_write_ops_test [id=21c86533f18c4c469d0e243959065a3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.404250 4378 tablet_service.cc:1467] Processing CreateTablet for tablet e530036caf1c4eb2ad2356427dd66caf (DEFAULT_TABLE table=txn_write_ops_test [id=21c86533f18c4c469d0e243959065a3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.404896 4239 tablet_service.cc:1467] Processing CreateTablet for tablet 5834112464e9439196fe6786cccc3b21 (DEFAULT_TABLE table=txn_write_ops_test [id=21c86533f18c4c469d0e243959065a3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.405144 4240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e530036caf1c4eb2ad2356427dd66caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.404879 4105 tablet_service.cc:1467] Processing CreateTablet for tablet e530036caf1c4eb2ad2356427dd66caf (DEFAULT_TABLE table=txn_write_ops_test [id=21c86533f18c4c469d0e243959065a3c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.405608 4239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5834112464e9439196fe6786cccc3b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.406003 4105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e530036caf1c4eb2ad2356427dd66caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.406978 4171 heartbeater.cc:502] Master 127.0.104.62:33533 was elected leader, sending a full tablet report...
I20250112 20:51:47.404232 4377 tablet_service.cc:1467] Processing CreateTablet for tablet 5834112464e9439196fe6786cccc3b21 (DEFAULT_TABLE table=txn_write_ops_test [id=21c86533f18c4c469d0e243959065a3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.407012 4378 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e530036caf1c4eb2ad2356427dd66caf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.407980 4103 tablet_service.cc:1467] Processing CreateTablet for tablet 5834112464e9439196fe6786cccc3b21 (DEFAULT_TABLE table=txn_write_ops_test [id=21c86533f18c4c469d0e243959065a3c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:47.408493 4103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5834112464e9439196fe6786cccc3b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.409332 4377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5834112464e9439196fe6786cccc3b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.424172 4466 tablet_bootstrap.cc:492] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e: Bootstrap starting.
I20250112 20:51:47.424889 4467 tablet_bootstrap.cc:492] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a: Bootstrap starting.
I20250112 20:51:47.425837 4465 tablet_bootstrap.cc:492] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef: Bootstrap starting.
I20250112 20:51:47.427642 4466 tablet_bootstrap.cc:654] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.428316 4467 tablet_bootstrap.cc:654] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.428962 4466 log.cc:826] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:47.429212 4465 tablet_bootstrap.cc:654] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.429600 4467 log.cc:826] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:47.430419 4465 log.cc:826] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:47.434911 4465 tablet_bootstrap.cc:492] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef: No bootstrap required, opened a new log
I20250112 20:51:47.435041 4467 tablet_bootstrap.cc:492] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a: No bootstrap required, opened a new log
I20250112 20:51:47.435145 4466 tablet_bootstrap.cc:492] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e: No bootstrap required, opened a new log
I20250112 20:51:47.435216 4465 ts_tablet_manager.cc:1397] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20250112 20:51:47.435285 4467 ts_tablet_manager.cc:1397] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250112 20:51:47.435438 4466 ts_tablet_manager.cc:1397] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250112 20:51:47.441905 4466 raft_consensus.cc:357] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.442255 4466 raft_consensus.cc:383] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.442353 4466 raft_consensus.cc:738] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247cc258515140bcb8d780b06c1f608e, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.442994 4466 consensus_queue.cc:260] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.444069 4307 heartbeater.cc:502] Master 127.0.104.62:33533 was elected leader, sending a full tablet report...
I20250112 20:51:47.444077 4465 raft_consensus.cc:357] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.444458 4465 raft_consensus.cc:383] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.444561 4466 ts_tablet_manager.cc:1428] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20250112 20:51:47.444586 4465 raft_consensus.cc:738] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4144e9bc5a0844bd82425be2d44e32ef, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.444978 4466 tablet_bootstrap.cc:492] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e: Bootstrap starting.
I20250112 20:51:47.445174 4465 consensus_queue.cc:260] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.446725 4466 tablet_bootstrap.cc:654] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.446797 4465 ts_tablet_manager.cc:1428] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20250112 20:51:47.447160 4465 tablet_bootstrap.cc:492] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef: Bootstrap starting.
I20250112 20:51:47.448935 4465 tablet_bootstrap.cc:654] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.448673 4467 raft_consensus.cc:357] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.449087 4467 raft_consensus.cc:383] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.449193 4467 raft_consensus.cc:738] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75305c00378d416ea46c1a8b59d7337a, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.452927 4467 consensus_queue.cc:260] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.454056 4448 heartbeater.cc:502] Master 127.0.104.62:33533 was elected leader, sending a full tablet report...
I20250112 20:51:47.454651 4467 ts_tablet_manager.cc:1428] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a: Time spent starting tablet: real 0.019s user 0.008s sys 0.005s
I20250112 20:51:47.455036 4467 tablet_bootstrap.cc:492] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a: Bootstrap starting.
I20250112 20:51:47.455935 4475 raft_consensus.cc:491] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.456162 4475 raft_consensus.cc:513] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.456857 4467 tablet_bootstrap.cc:654] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:47.458957 4465 tablet_bootstrap.cc:492] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef: No bootstrap required, opened a new log
I20250112 20:51:47.459131 4465 ts_tablet_manager.cc:1397] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20250112 20:51:47.460062 4465 raft_consensus.cc:357] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.460253 4465 raft_consensus.cc:383] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.460341 4465 raft_consensus.cc:738] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4144e9bc5a0844bd82425be2d44e32ef, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.460635 4465 consensus_queue.cc:260] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.461230 4465 ts_tablet_manager.cc:1428] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:47.462759 4466 tablet_bootstrap.cc:492] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e: No bootstrap required, opened a new log
I20250112 20:51:47.462983 4466 ts_tablet_manager.cc:1397] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.008s
I20250112 20:51:47.463898 4466 raft_consensus.cc:357] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.464166 4466 raft_consensus.cc:383] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.464383 4466 raft_consensus.cc:738] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247cc258515140bcb8d780b06c1f608e, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.464701 4466 consensus_queue.cc:260] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.465464 4466 ts_tablet_manager.cc:1428] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:47.466814 4475 leader_election.cc:290] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4144e9bc5a0844bd82425be2d44e32ef (127.0.104.1:39813), 247cc258515140bcb8d780b06c1f608e (127.0.104.2:41359)
I20250112 20:51:47.468466 4126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834112464e9439196fe6786cccc3b21" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4144e9bc5a0844bd82425be2d44e32ef" is_pre_election: true
I20250112 20:51:47.469135 4126 raft_consensus.cc:2463] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 0.
I20250112 20:51:47.470423 4332 leader_election.cc:304] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4144e9bc5a0844bd82425be2d44e32ef, 75305c00378d416ea46c1a8b59d7337a; no voters:
I20250112 20:51:47.471254 4467 tablet_bootstrap.cc:492] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a: No bootstrap required, opened a new log
I20250112 20:51:47.471395 4475 raft_consensus.cc:2798] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.471549 4467 ts_tablet_manager.cc:1397] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20250112 20:51:47.471583 4475 raft_consensus.cc:491] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.471724 4475 raft_consensus.cc:3054] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.472470 4467 raft_consensus.cc:357] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.472718 4467 raft_consensus.cc:383] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:47.472798 4467 raft_consensus.cc:738] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75305c00378d416ea46c1a8b59d7337a, State: Initialized, Role: FOLLOWER
I20250112 20:51:47.473001 4467 consensus_queue.cc:260] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.473832 4467 ts_tablet_manager.cc:1428] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:47.474252 4257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834112464e9439196fe6786cccc3b21" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247cc258515140bcb8d780b06c1f608e" is_pre_election: true
I20250112 20:51:47.474759 4257 raft_consensus.cc:2463] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 0.
I20250112 20:51:47.474850 4475 raft_consensus.cc:513] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.475492 4475 leader_election.cc:290] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 election: Requested vote from peers 4144e9bc5a0844bd82425be2d44e32ef (127.0.104.1:39813), 247cc258515140bcb8d780b06c1f608e (127.0.104.2:41359)
I20250112 20:51:47.476044 4257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834112464e9439196fe6786cccc3b21" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247cc258515140bcb8d780b06c1f608e"
I20250112 20:51:47.476047 4126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5834112464e9439196fe6786cccc3b21" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4144e9bc5a0844bd82425be2d44e32ef"
I20250112 20:51:47.476225 4257 raft_consensus.cc:3054] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.476229 4126 raft_consensus.cc:3054] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.479264 4126 raft_consensus.cc:2463] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 1.
I20250112 20:51:47.479264 4257 raft_consensus.cc:2463] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 1.
I20250112 20:51:47.479630 4332 leader_election.cc:304] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4144e9bc5a0844bd82425be2d44e32ef, 75305c00378d416ea46c1a8b59d7337a; no voters:
I20250112 20:51:47.479910 4475 raft_consensus.cc:2798] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.480060 4475 raft_consensus.cc:695] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [term 1 LEADER]: Becoming Leader. State: Replica: 75305c00378d416ea46c1a8b59d7337a, State: Running, Role: LEADER
I20250112 20:51:47.480468 4475 consensus_queue.cc:237] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.485014 3962 catalog_manager.cc:5526] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a reported cstate change: term changed from 0 to 1, leader changed from <none> to 75305c00378d416ea46c1a8b59d7337a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "75305c00378d416ea46c1a8b59d7337a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:47.574138 4471 raft_consensus.cc:491] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.574344 4471 raft_consensus.cc:513] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.576102 4471 leader_election.cc:290] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4144e9bc5a0844bd82425be2d44e32ef (127.0.104.1:39813), 75305c00378d416ea46c1a8b59d7337a (127.0.104.3:43451)
I20250112 20:51:47.578424 4475 raft_consensus.cc:491] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:47.578652 4475 raft_consensus.cc:513] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.579316 4475 leader_election.cc:290] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4144e9bc5a0844bd82425be2d44e32ef (127.0.104.1:39813), 247cc258515140bcb8d780b06c1f608e (127.0.104.2:41359)
I20250112 20:51:47.579798 4126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530036caf1c4eb2ad2356427dd66caf" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4144e9bc5a0844bd82425be2d44e32ef" is_pre_election: true
I20250112 20:51:47.579890 4257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530036caf1c4eb2ad2356427dd66caf" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247cc258515140bcb8d780b06c1f608e" is_pre_election: true
I20250112 20:51:47.580085 4257 raft_consensus.cc:2463] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 0.
I20250112 20:51:47.580075 4126 raft_consensus.cc:2463] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 0.
I20250112 20:51:47.580480 4332 leader_election.cc:304] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4144e9bc5a0844bd82425be2d44e32ef, 75305c00378d416ea46c1a8b59d7337a; no voters:
I20250112 20:51:47.580932 4475 raft_consensus.cc:2798] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:47.581025 4475 raft_consensus.cc:491] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:47.581130 4475 raft_consensus.cc:3054] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.583426 4475 raft_consensus.cc:513] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.584081 4475 leader_election.cc:290] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 election: Requested vote from peers 4144e9bc5a0844bd82425be2d44e32ef (127.0.104.1:39813), 247cc258515140bcb8d780b06c1f608e (127.0.104.2:41359)
I20250112 20:51:47.584722 4257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530036caf1c4eb2ad2356427dd66caf" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247cc258515140bcb8d780b06c1f608e"
I20250112 20:51:47.584764 4126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530036caf1c4eb2ad2356427dd66caf" candidate_uuid: "75305c00378d416ea46c1a8b59d7337a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4144e9bc5a0844bd82425be2d44e32ef"
I20250112 20:51:47.584995 4257 raft_consensus.cc:3054] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.585000 4126 raft_consensus.cc:3054] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:47.585026 4125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530036caf1c4eb2ad2356427dd66caf" candidate_uuid: "247cc258515140bcb8d780b06c1f608e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4144e9bc5a0844bd82425be2d44e32ef" is_pre_election: true
I20250112 20:51:47.585088 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e530036caf1c4eb2ad2356427dd66caf" candidate_uuid: "247cc258515140bcb8d780b06c1f608e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75305c00378d416ea46c1a8b59d7337a" is_pre_election: true
I20250112 20:51:47.585523 4403 raft_consensus.cc:2388] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 247cc258515140bcb8d780b06c1f608e in current term 1: Already voted for candidate 75305c00378d416ea46c1a8b59d7337a in this term.
I20250112 20:51:47.586460 4191 leader_election.cc:304] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 247cc258515140bcb8d780b06c1f608e; no voters: 4144e9bc5a0844bd82425be2d44e32ef, 75305c00378d416ea46c1a8b59d7337a
I20250112 20:51:47.587872 4257 raft_consensus.cc:2463] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 1.
I20250112 20:51:47.587873 4126 raft_consensus.cc:2463] T e530036caf1c4eb2ad2356427dd66caf P 4144e9bc5a0844bd82425be2d44e32ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75305c00378d416ea46c1a8b59d7337a in term 1.
I20250112 20:51:47.588048 4471 raft_consensus.cc:2743] T e530036caf1c4eb2ad2356427dd66caf P 247cc258515140bcb8d780b06c1f608e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250112 20:51:47.588326 4330 leader_election.cc:304] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247cc258515140bcb8d780b06c1f608e, 75305c00378d416ea46c1a8b59d7337a; no voters:
I20250112 20:51:47.588688 4475 raft_consensus.cc:2798] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:47.588951 4475 raft_consensus.cc:695] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [term 1 LEADER]: Becoming Leader. State: Replica: 75305c00378d416ea46c1a8b59d7337a, State: Running, Role: LEADER
I20250112 20:51:47.589238 4475 consensus_queue.cc:237] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:47.591712 3962 catalog_manager.cc:5526] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a reported cstate change: term changed from 0 to 1, leader changed from <none> to 75305c00378d416ea46c1a8b59d7337a (127.0.104.3). New cstate: current_term: 1 leader_uuid: "75305c00378d416ea46c1a8b59d7337a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:47.594655 4449 tablet.cc:2367] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:47.611164 4308 tablet.cc:2367] T 5834112464e9439196fe6786cccc3b21 P 247cc258515140bcb8d780b06c1f608e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:47.648612 4172 tablet.cc:2367] T 5834112464e9439196fe6786cccc3b21 P 4144e9bc5a0844bd82425be2d44e32ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:47.732962 4484 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:51:47.976011 3962 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44676:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:47.976699 3962 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:47.982836 4377 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3090917bc740a59b0211e581964f41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb9949bd888b4778889ef1d92f8d0726]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:47.982645 4103 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3090917bc740a59b0211e581964f41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb9949bd888b4778889ef1d92f8d0726]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:47.982753 4240 tablet_service.cc:1467] Processing CreateTablet for tablet 2b3090917bc740a59b0211e581964f41 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fb9949bd888b4778889ef1d92f8d0726]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:47.983613 4377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3090917bc740a59b0211e581964f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.983614 4240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3090917bc740a59b0211e581964f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.983616 4103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b3090917bc740a59b0211e581964f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:47.988183 4475 consensus_queue.cc:1035] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.989029 4475 consensus_queue.cc:1035] T 5834112464e9439196fe6786cccc3b21 P 75305c00378d416ea46c1a8b59d7337a [LEADER]: Connected to new peer: Peer: permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:47.999522 4529 tablet_bootstrap.cc:492] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: Bootstrap starting.
I20250112 20:51:48.003288 4529 tablet_bootstrap.cc:654] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.004449 4537 tablet_bootstrap.cc:492] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: Bootstrap starting.
I20250112 20:51:48.005815 4529 tablet_bootstrap.cc:492] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: No bootstrap required, opened a new log
I20250112 20:51:48.005954 4529 ts_tablet_manager.cc:1397] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250112 20:51:48.006845 4537 tablet_bootstrap.cc:654] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.006760 4529 raft_consensus.cc:357] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.007005 4529 raft_consensus.cc:383] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.007103 4529 raft_consensus.cc:738] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247cc258515140bcb8d780b06c1f608e, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.008755 4529 consensus_queue.cc:260] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.008955 4537 tablet_bootstrap.cc:492] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: No bootstrap required, opened a new log
I20250112 20:51:48.009101 4537 ts_tablet_manager.cc:1397] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:48.009425 4471 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.009728 4471 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:48.009886 4537 raft_consensus.cc:357] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.010071 4537 raft_consensus.cc:383] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.010152 4537 raft_consensus.cc:738] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75305c00378d416ea46c1a8b59d7337a, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.010417 4529 ts_tablet_manager.cc:1428] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:48.010823 4537 consensus_queue.cc:260] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.011373 4477 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.011639 4477 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:48.013231 4537 ts_tablet_manager.cc:1428] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:48.013391 4530 tablet_bootstrap.cc:492] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: Bootstrap starting.
I20250112 20:51:48.015174 4530 tablet_bootstrap.cc:654] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:48.016872 4530 tablet_bootstrap.cc:492] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: No bootstrap required, opened a new log
I20250112 20:51:48.016979 4530 ts_tablet_manager.cc:1397] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:48.017488 4530 raft_consensus.cc:357] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.017653 4530 raft_consensus.cc:383] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:48.017715 4530 raft_consensus.cc:738] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4144e9bc5a0844bd82425be2d44e32ef, State: Initialized, Role: FOLLOWER
I20250112 20:51:48.017896 4530 consensus_queue.cc:260] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.018326 4530 ts_tablet_manager.cc:1428] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:48.018257 4528 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.018572 4528 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:48.074506 4477 consensus_queue.cc:1035] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [LEADER]: Connected to new peer: Peer: permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.078778 4477 consensus_queue.cc:1035] T e530036caf1c4eb2ad2356427dd66caf P 75305c00378d416ea46c1a8b59d7337a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.220345 4528 raft_consensus.cc:491] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:48.220628 4528 raft_consensus.cc:513] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.222081 4528 leader_election.cc:290] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 247cc258515140bcb8d780b06c1f608e (127.0.104.2:41359), 75305c00378d416ea46c1a8b59d7337a (127.0.104.3:43451)
I20250112 20:51:48.228854 4257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3090917bc740a59b0211e581964f41" candidate_uuid: "4144e9bc5a0844bd82425be2d44e32ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247cc258515140bcb8d780b06c1f608e" is_pre_election: true
I20250112 20:51:48.229066 4257 raft_consensus.cc:2463] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4144e9bc5a0844bd82425be2d44e32ef in term 0.
I20250112 20:51:48.229200 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3090917bc740a59b0211e581964f41" candidate_uuid: "4144e9bc5a0844bd82425be2d44e32ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75305c00378d416ea46c1a8b59d7337a" is_pre_election: true
I20250112 20:51:48.229430 4403 raft_consensus.cc:2463] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4144e9bc5a0844bd82425be2d44e32ef in term 0.
I20250112 20:51:48.229586 4053 leader_election.cc:304] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247cc258515140bcb8d780b06c1f608e, 4144e9bc5a0844bd82425be2d44e32ef; no voters:
I20250112 20:51:48.229964 4528 raft_consensus.cc:2798] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:48.230089 4528 raft_consensus.cc:491] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:48.230201 4528 raft_consensus.cc:3054] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.232069 4528 raft_consensus.cc:513] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.232609 4528 leader_election.cc:290] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [CANDIDATE]: Term 1 election: Requested vote from peers 247cc258515140bcb8d780b06c1f608e (127.0.104.2:41359), 75305c00378d416ea46c1a8b59d7337a (127.0.104.3:43451)
I20250112 20:51:48.233029 4257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3090917bc740a59b0211e581964f41" candidate_uuid: "4144e9bc5a0844bd82425be2d44e32ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247cc258515140bcb8d780b06c1f608e"
I20250112 20:51:48.233044 4403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b3090917bc740a59b0211e581964f41" candidate_uuid: "4144e9bc5a0844bd82425be2d44e32ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75305c00378d416ea46c1a8b59d7337a"
I20250112 20:51:48.233189 4403 raft_consensus.cc:3054] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.233181 4257 raft_consensus.cc:3054] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:48.234898 4257 raft_consensus.cc:2463] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4144e9bc5a0844bd82425be2d44e32ef in term 1.
I20250112 20:51:48.235023 4403 raft_consensus.cc:2463] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4144e9bc5a0844bd82425be2d44e32ef in term 1.
I20250112 20:51:48.235307 4053 leader_election.cc:304] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 247cc258515140bcb8d780b06c1f608e, 4144e9bc5a0844bd82425be2d44e32ef; no voters:
I20250112 20:51:48.235651 4528 raft_consensus.cc:2798] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:48.236040 4528 raft_consensus.cc:695] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [term 1 LEADER]: Becoming Leader. State: Replica: 4144e9bc5a0844bd82425be2d44e32ef, State: Running, Role: LEADER
I20250112 20:51:48.236429 4528 consensus_queue.cc:237] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } }
I20250112 20:51:48.238263 4546 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: TxnStatusTablet state changed. Reason: New leader 4144e9bc5a0844bd82425be2d44e32ef. Latest consensus state: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.238598 4546 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:48.239246 4548 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:48.239483 4548 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:48.239934 3963 catalog_manager.cc:5526] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 4144e9bc5a0844bd82425be2d44e32ef (127.0.104.1). New cstate: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:48.668769 4257 raft_consensus.cc:1270] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e [term 1 FOLLOWER]: Refusing update from remote peer 4144e9bc5a0844bd82425be2d44e32ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:48.669104 4403 raft_consensus.cc:1270] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a [term 1 FOLLOWER]: Refusing update from remote peer 4144e9bc5a0844bd82425be2d44e32ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:48.669353 4528 consensus_queue.cc:1035] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.669735 4546 consensus_queue.cc:1035] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:48.677335 4550 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: TxnStatusTablet state changed. Reason: New leader 4144e9bc5a0844bd82425be2d44e32ef. Latest consensus state: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.677699 4550 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:48.677464 4555 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: TxnStatusTablet state changed. Reason: New leader 4144e9bc5a0844bd82425be2d44e32ef. Latest consensus state: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.677944 4555 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:48.679591 4560 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.680011 4560 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:48.683267 4554 mvcc.cc:204] Tried to move back new op lower bound from 7113585085099667456 to 7113585083339804672. Current Snapshot: MvccSnapshot[applied={T|T < 7113585085099667456}]
I20250112 20:51:48.683959 4555 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.684217 4555 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 247cc258515140bcb8d780b06c1f608e: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:48.684715 4556 mvcc.cc:204] Tried to move back new op lower bound from 7113585085099667456 to 7113585083339804672. Current Snapshot: MvccSnapshot[applied={T|T < 7113585085099667456}]
I20250112 20:51:48.687029 4550 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.687228 4550 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 75305c00378d416ea46c1a8b59d7337a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:48.689204 4528 tablet_replica.cc:440] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4144e9bc5a0844bd82425be2d44e32ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4144e9bc5a0844bd82425be2d44e32ef" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39813 } } peers { permanent_uuid: "247cc258515140bcb8d780b06c1f608e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41359 } } peers { permanent_uuid: "75305c00378d416ea46c1a8b59d7337a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43451 } } }
I20250112 20:51:48.689435 4528 tablet_replica.cc:443] T 2b3090917bc740a59b0211e581964f41 P 4144e9bc5a0844bd82425be2d44e32ef: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:49.927572 416 txn_write_ops-itest.cc:831] txn write RPCs completed: 8
I20250112 20:51:49.927685 416 txn_write_ops-itest.cc:833] txn write RPC rate: 160 req/sec
I20250112 20:51:49.954025 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 4037
I20250112 20:51:49.955013 4166 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:50.085119 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 4037
I20250112 20:51:50.110280 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 4173
I20250112 20:51:50.111428 4302 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:50.210314 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 4173
I20250112 20:51:50.229310 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 4309
I20250112 20:51:50.230379 4443 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:50.365949 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 4309
I20250112 20:51:50.387276 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 3932
I20250112 20:51:50.388275 4003 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:50.497221 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 3932
2025-01-12T20:51:50Z chronyd exiting
[ OK ] TxnWriteOpsITest.WriteOpPerf (5156 ms)
[ RUN ] TxnWriteOpsITest.WriteOpForNonExistentTxn
2025-01-12T20:51:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:50Z Disabled control of system clock
I20250112 20:51:50.566810 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44305
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44305
--txn_manager_enabled=true with env {}
W20250112 20:51:50.877382 4638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:50.877681 4638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:50.877765 4638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:50.885131 4638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:50.885231 4638 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:50.885267 4638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:50.885322 4638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:50.885367 4638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:50.894343 4638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44305
--txn_manager_enabled=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:50.895802 4638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:50.897454 4638 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:50.907131 4643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:50.907228 4644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:50.907665 4646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:50.907958 4638 server_base.cc:1034] running on GCE node
I20250112 20:51:50.908667 4638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:50.909893 4638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:50.911101 4638 hybrid_clock.cc:648] HybridClock initialized: now 1736715110911059 us; error 37 us; skew 500 ppm
I20250112 20:51:50.911428 4638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:50.917546 4638 webserver.cc:458] Webserver started at http://127.0.104.62:34453/ using document root <none> and password file <none>
I20250112 20:51:50.918291 4638 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:50.918396 4638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:50.918713 4638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:50.921242 4638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "d6dec72b10154eca8ff0a32158154736"
format_stamp: "Formatted at 2025-01-12 20:51:50 on dist-test-slave-3gql"
I20250112 20:51:50.921960 4638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "d6dec72b10154eca8ff0a32158154736"
format_stamp: "Formatted at 2025-01-12 20:51:50 on dist-test-slave-3gql"
I20250112 20:51:50.926666 4638 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250112 20:51:50.929972 4652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:50.931360 4638 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250112 20:51:50.931528 4638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "d6dec72b10154eca8ff0a32158154736"
format_stamp: "Formatted at 2025-01-12 20:51:50 on dist-test-slave-3gql"
I20250112 20:51:50.931730 4638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:50.950480 4638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:50.951366 4638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:50.951642 4638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:50.971235 4638 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44305
I20250112 20:51:50.971268 4713 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44305 every 8 connection(s)
I20250112 20:51:50.972478 4638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:50.973328 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 4638
I20250112 20:51:50.973553 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:50.977241 4714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:50.987318 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736: Bootstrap starting.
I20250112 20:51:50.990320 4714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:50.991410 4714 log.cc:826] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:50.993706 4714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736: No bootstrap required, opened a new log
I20250112 20:51:50.999354 4714 raft_consensus.cc:357] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dec72b10154eca8ff0a32158154736" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44305 } }
I20250112 20:51:50.999662 4714 raft_consensus.cc:383] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:50.999749 4714 raft_consensus.cc:738] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6dec72b10154eca8ff0a32158154736, State: Initialized, Role: FOLLOWER
I20250112 20:51:51.000317 4714 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dec72b10154eca8ff0a32158154736" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44305 } }
I20250112 20:51:51.000591 4714 raft_consensus.cc:397] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:51.000736 4714 raft_consensus.cc:491] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:51.000906 4714 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:51.002897 4714 raft_consensus.cc:513] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dec72b10154eca8ff0a32158154736" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44305 } }
I20250112 20:51:51.003331 4714 leader_election.cc:304] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6dec72b10154eca8ff0a32158154736; no voters:
I20250112 20:51:51.003901 4714 leader_election.cc:290] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:51.004019 4719 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:51.004617 4719 raft_consensus.cc:695] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [term 1 LEADER]: Becoming Leader. State: Replica: d6dec72b10154eca8ff0a32158154736, State: Running, Role: LEADER
I20250112 20:51:51.005159 4719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dec72b10154eca8ff0a32158154736" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44305 } }
I20250112 20:51:51.005776 4714 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:51.008283 4720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6dec72b10154eca8ff0a32158154736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dec72b10154eca8ff0a32158154736" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44305 } } }
I20250112 20:51:51.008388 4721 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6dec72b10154eca8ff0a32158154736. Latest consensus state: current_term: 1 leader_uuid: "d6dec72b10154eca8ff0a32158154736" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6dec72b10154eca8ff0a32158154736" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44305 } } }
I20250112 20:51:51.008713 4720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:51.008783 4721 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:51.009565 4724 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:51.016009 4724 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:51.028723 4724 catalog_manager.cc:1348] Generated new cluster ID: 3d128198ea9d45948f4d282d294e6cd4
I20250112 20:51:51.028861 4724 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:51.042062 4724 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:51.043253 4724 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:51.052263 4724 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d6dec72b10154eca8ff0a32158154736: Generated new TSK 0
I20250112 20:51:51.053090 4724 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:51.064321 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44305
--builtin_ntp_servers=127.0.104.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:51.411375 4738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:51.411697 4738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:51.411841 4738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:51.419225 4738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:51.419364 4738 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:51.419497 4738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:51.428418 4738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:44305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:51.430116 4738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:51.431759 4738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:51.442873 4743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:51.442903 4746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:51.443104 4744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:51.444217 4738 server_base.cc:1034] running on GCE node
I20250112 20:51:51.444823 4738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:51.445715 4738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:51.446914 4738 hybrid_clock.cc:648] HybridClock initialized: now 1736715111446874 us; error 37 us; skew 500 ppm
I20250112 20:51:51.447203 4738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:51.449405 4738 webserver.cc:458] Webserver started at http://127.0.104.1:44571/ using document root <none> and password file <none>
I20250112 20:51:51.450173 4738 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:51.450273 4738 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:51.450578 4738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:51.453087 4738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "920b0b46e8754068ab10e3cb31f8df0d"
format_stamp: "Formatted at 2025-01-12 20:51:51 on dist-test-slave-3gql"
I20250112 20:51:51.453814 4738 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "920b0b46e8754068ab10e3cb31f8df0d"
format_stamp: "Formatted at 2025-01-12 20:51:51 on dist-test-slave-3gql"
I20250112 20:51:51.458592 4738 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250112 20:51:51.461642 4752 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:51.462839 4738 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250112 20:51:51.462972 4738 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "920b0b46e8754068ab10e3cb31f8df0d"
format_stamp: "Formatted at 2025-01-12 20:51:51 on dist-test-slave-3gql"
I20250112 20:51:51.463138 4738 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:51.480898 4738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:51.481707 4738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:51.481963 4738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:51.484759 4738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:51.484850 4738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:51.484989 4738 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:51.485060 4738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:51.528658 4738 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:41759
I20250112 20:51:51.528692 4871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:41759 every 8 connection(s)
I20250112 20:51:51.529932 4738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:51.531884 4872 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44305
I20250112 20:51:51.532101 4872 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:51.532848 4872 heartbeater.cc:510] Master 127.0.104.62:44305 requested a full tablet report, sending...
I20250112 20:51:51.534762 4669 ts_manager.cc:194] Registered new tserver with Master: 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1:41759)
I20250112 20:51:51.536207 4669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:34019
I20250112 20:51:51.537003 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 4738
I20250112 20:51:51.537174 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:51.540024 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44305
--builtin_ntp_servers=127.0.104.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:51.873557 4874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:51.873854 4874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:51.873945 4874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:51.881363 4874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:51.881500 4874 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:51.881603 4874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:51.890309 4874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:44305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:51.891779 4874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:51.893373 4874 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:51.904326 4880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:51.904381 4879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:51.904358 4882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:51.905288 4874 server_base.cc:1034] running on GCE node
I20250112 20:51:51.905905 4874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:51.906828 4874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:51.908080 4874 hybrid_clock.cc:648] HybridClock initialized: now 1736715111908033 us; error 41 us; skew 500 ppm
I20250112 20:51:51.908421 4874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:51.910637 4874 webserver.cc:458] Webserver started at http://127.0.104.2:45691/ using document root <none> and password file <none>
I20250112 20:51:51.911350 4874 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:51.911541 4874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:51.911859 4874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:51.915897 4874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "70442fbead15433dbde12a2dcbb850f0"
format_stamp: "Formatted at 2025-01-12 20:51:51 on dist-test-slave-3gql"
I20250112 20:51:51.916682 4874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "70442fbead15433dbde12a2dcbb850f0"
format_stamp: "Formatted at 2025-01-12 20:51:51 on dist-test-slave-3gql"
I20250112 20:51:51.921710 4874 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20250112 20:51:51.924909 4888 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:51.926205 4874 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250112 20:51:51.926373 4874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "70442fbead15433dbde12a2dcbb850f0"
format_stamp: "Formatted at 2025-01-12 20:51:51 on dist-test-slave-3gql"
I20250112 20:51:51.926587 4874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:51.950438 4874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:51.951277 4874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:51.951551 4874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:51.954288 4874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:51.954393 4874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:51.954531 4874 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:51.954617 4874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:52.004509 4874 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42203
I20250112 20:51:52.004531 5007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42203 every 8 connection(s)
I20250112 20:51:52.005942 4874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:52.007997 5008 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44305
I20250112 20:51:52.008203 5008 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:52.009001 5008 heartbeater.cc:510] Master 127.0.104.62:44305 requested a full tablet report, sending...
I20250112 20:51:52.009894 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 4874
I20250112 20:51:52.010124 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:52.010368 4669 ts_manager.cc:194] Registered new tserver with Master: 70442fbead15433dbde12a2dcbb850f0 (127.0.104.2:42203)
I20250112 20:51:52.011318 4669 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:35129
I20250112 20:51:52.013654 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44305
--builtin_ntp_servers=127.0.104.20:44911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:52.331205 5010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:52.331475 5010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:52.331609 5010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:52.338968 5010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:52.339119 5010 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:52.339255 5010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:52.348155 5010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:44305
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:52.349679 5010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:52.351287 5010 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:52.362107 5016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:52.362031 5015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:52.362054 5018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:52.363168 5010 server_base.cc:1034] running on GCE node
I20250112 20:51:52.363729 5010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:52.364635 5010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:52.365847 5010 hybrid_clock.cc:648] HybridClock initialized: now 1736715112365803 us; error 34 us; skew 500 ppm
I20250112 20:51:52.366137 5010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:52.368276 5010 webserver.cc:458] Webserver started at http://127.0.104.3:43643/ using document root <none> and password file <none>
I20250112 20:51:52.369016 5010 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:52.369148 5010 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:52.369478 5010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:52.371968 5010 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "e6e5e1473a414d25a49e3c221025b2ed"
format_stamp: "Formatted at 2025-01-12 20:51:52 on dist-test-slave-3gql"
I20250112 20:51:52.372777 5010 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "e6e5e1473a414d25a49e3c221025b2ed"
format_stamp: "Formatted at 2025-01-12 20:51:52 on dist-test-slave-3gql"
I20250112 20:51:52.377665 5010 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:52.380860 5024 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:52.382077 5010 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250112 20:51:52.382220 5010 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "e6e5e1473a414d25a49e3c221025b2ed"
format_stamp: "Formatted at 2025-01-12 20:51:52 on dist-test-slave-3gql"
I20250112 20:51:52.382395 5010 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:52.402509 5010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:52.403275 5010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:52.403522 5010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:52.406262 5010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:52.406355 5010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:52.406510 5010 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:52.406584 5010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:52.454348 5010 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:37583
I20250112 20:51:52.454365 5143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:37583 every 8 connection(s)
I20250112 20:51:52.456139 5010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:52.459388 5144 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44305
I20250112 20:51:52.459610 5144 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:52.460347 5144 heartbeater.cc:510] Master 127.0.104.62:44305 requested a full tablet report, sending...
I20250112 20:51:52.461752 4668 ts_manager.cc:194] Registered new tserver with Master: e6e5e1473a414d25a49e3c221025b2ed (127.0.104.3:37583)
I20250112 20:51:52.462491 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 5010
I20250112 20:51:52.462668 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.WriteOpForNonExistentTxn.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:52.462957 4668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:45469
I20250112 20:51:52.466419 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:52.492132 4668 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53964:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:52.503831 4668 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:52.538733 4872 heartbeater.cc:502] Master 127.0.104.62:44305 was elected leader, sending a full tablet report...
I20250112 20:51:52.558933 4806 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea40a8ab6594cbeb6681d9aca4138a3 (DEFAULT_TABLE table=txn_write_ops_test [id=8c1b00783b9f4938af7b593658901c0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.560585 4804 tablet_service.cc:1467] Processing CreateTablet for tablet 2e02a0f9316945c792d84def95b67283 (DEFAULT_TABLE table=txn_write_ops_test [id=8c1b00783b9f4938af7b593658901c0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.561584 4804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e02a0f9316945c792d84def95b67283. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.562497 5078 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea40a8ab6594cbeb6681d9aca4138a3 (DEFAULT_TABLE table=txn_write_ops_test [id=8c1b00783b9f4938af7b593658901c0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.563805 5078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea40a8ab6594cbeb6681d9aca4138a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.565002 4942 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea40a8ab6594cbeb6681d9aca4138a3 (DEFAULT_TABLE table=txn_write_ops_test [id=8c1b00783b9f4938af7b593658901c0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.566256 4806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea40a8ab6594cbeb6681d9aca4138a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.566330 4942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ea40a8ab6594cbeb6681d9aca4138a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.568023 5077 tablet_service.cc:1467] Processing CreateTablet for tablet 2e02a0f9316945c792d84def95b67283 (DEFAULT_TABLE table=txn_write_ops_test [id=8c1b00783b9f4938af7b593658901c0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.568473 5077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e02a0f9316945c792d84def95b67283. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.568521 4941 tablet_service.cc:1467] Processing CreateTablet for tablet 2e02a0f9316945c792d84def95b67283 (DEFAULT_TABLE table=txn_write_ops_test [id=8c1b00783b9f4938af7b593658901c0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:52.569018 4941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e02a0f9316945c792d84def95b67283. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:52.585759 5162 tablet_bootstrap.cc:492] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d: Bootstrap starting.
I20250112 20:51:52.589471 5163 tablet_bootstrap.cc:492] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed: Bootstrap starting.
I20250112 20:51:52.592819 5163 tablet_bootstrap.cc:654] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.595320 5161 tablet_bootstrap.cc:492] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0: Bootstrap starting.
I20250112 20:51:52.597263 5162 tablet_bootstrap.cc:654] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.598080 5161 tablet_bootstrap.cc:654] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.598512 5162 log.cc:826] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:52.599185 5161 log.cc:826] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:52.599385 5163 log.cc:826] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:52.601720 5163 tablet_bootstrap.cc:492] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed: No bootstrap required, opened a new log
I20250112 20:51:52.601959 5163 ts_tablet_manager.cc:1397] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20250112 20:51:52.602329 5162 tablet_bootstrap.cc:492] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d: No bootstrap required, opened a new log
I20250112 20:51:52.602648 5162 ts_tablet_manager.cc:1397] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20250112 20:51:52.606240 5161 tablet_bootstrap.cc:492] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0: No bootstrap required, opened a new log
I20250112 20:51:52.606607 5161 ts_tablet_manager.cc:1397] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.006s
I20250112 20:51:52.609361 5163 raft_consensus.cc:357] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.609791 5163 raft_consensus.cc:383] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.609925 5163 raft_consensus.cc:738] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e5e1473a414d25a49e3c221025b2ed, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.610606 5163 consensus_queue.cc:260] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.613433 5162 raft_consensus.cc:357] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.613950 5162 raft_consensus.cc:383] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.614096 5162 raft_consensus.cc:738] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920b0b46e8754068ab10e3cb31f8df0d, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.614893 5162 consensus_queue.cc:260] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.618747 5161 raft_consensus.cc:357] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.619306 5161 raft_consensus.cc:383] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.619468 5161 raft_consensus.cc:738] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70442fbead15433dbde12a2dcbb850f0, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.620357 5161 consensus_queue.cc:260] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.621611 5144 heartbeater.cc:502] Master 127.0.104.62:44305 was elected leader, sending a full tablet report...
I20250112 20:51:52.622733 5163 ts_tablet_manager.cc:1428] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed: Time spent starting tablet: real 0.021s user 0.013s sys 0.007s
I20250112 20:51:52.623222 5163 tablet_bootstrap.cc:492] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed: Bootstrap starting.
I20250112 20:51:52.625380 5163 tablet_bootstrap.cc:654] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.625867 5161 ts_tablet_manager.cc:1428] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0: Time spent starting tablet: real 0.019s user 0.015s sys 0.000s
I20250112 20:51:52.626224 5008 heartbeater.cc:502] Master 127.0.104.62:44305 was elected leader, sending a full tablet report...
I20250112 20:51:52.628880 5161 tablet_bootstrap.cc:492] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0: Bootstrap starting.
I20250112 20:51:52.631274 5161 tablet_bootstrap.cc:654] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.634418 5162 ts_tablet_manager.cc:1428] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d: Time spent starting tablet: real 0.032s user 0.014s sys 0.006s
I20250112 20:51:52.636101 5162 tablet_bootstrap.cc:492] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d: Bootstrap starting.
I20250112 20:51:52.636919 5161 tablet_bootstrap.cc:492] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0: No bootstrap required, opened a new log
I20250112 20:51:52.637115 5161 ts_tablet_manager.cc:1397] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20250112 20:51:52.638370 5162 tablet_bootstrap.cc:654] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:52.639168 5161 raft_consensus.cc:357] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.639508 5161 raft_consensus.cc:383] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.639657 5161 raft_consensus.cc:738] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70442fbead15433dbde12a2dcbb850f0, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.640059 5161 consensus_queue.cc:260] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.641471 5161 ts_tablet_manager.cc:1428] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:52.643106 5163 tablet_bootstrap.cc:492] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed: No bootstrap required, opened a new log
I20250112 20:51:52.643350 5163 ts_tablet_manager.cc:1397] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.012s
I20250112 20:51:52.644474 5163 raft_consensus.cc:357] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.644887 5163 raft_consensus.cc:383] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.645020 5163 raft_consensus.cc:738] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e5e1473a414d25a49e3c221025b2ed, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.645316 5163 consensus_queue.cc:260] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.646723 5163 ts_tablet_manager.cc:1428] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:52.647719 5162 tablet_bootstrap.cc:492] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d: No bootstrap required, opened a new log
I20250112 20:51:52.647862 5162 ts_tablet_manager.cc:1397] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250112 20:51:52.648450 5162 raft_consensus.cc:357] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.648641 5162 raft_consensus.cc:383] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:52.648748 5162 raft_consensus.cc:738] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920b0b46e8754068ab10e3cb31f8df0d, State: Initialized, Role: FOLLOWER
I20250112 20:51:52.648949 5162 consensus_queue.cc:260] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.649317 5162 ts_tablet_manager.cc:1428] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250112 20:51:52.686316 5168 raft_consensus.cc:491] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.686674 5168 raft_consensus.cc:513] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.689138 5168 leader_election.cc:290] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6e5e1473a414d25a49e3c221025b2ed (127.0.104.3:37583), 70442fbead15433dbde12a2dcbb850f0 (127.0.104.2:42203)
I20250112 20:51:52.696904 4962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea40a8ab6594cbeb6681d9aca4138a3" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70442fbead15433dbde12a2dcbb850f0" is_pre_election: true
I20250112 20:51:52.697453 4962 raft_consensus.cc:2463] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 0.
I20250112 20:51:52.698055 4755 leader_election.cc:304] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70442fbead15433dbde12a2dcbb850f0, 920b0b46e8754068ab10e3cb31f8df0d; no voters:
I20250112 20:51:52.698498 5168 raft_consensus.cc:2798] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.698474 5099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea40a8ab6594cbeb6681d9aca4138a3" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6e5e1473a414d25a49e3c221025b2ed" is_pre_election: true
I20250112 20:51:52.698666 5168 raft_consensus.cc:491] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.698770 5168 raft_consensus.cc:3054] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.698887 5099 raft_consensus.cc:2463] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 0.
I20250112 20:51:52.702330 5168 raft_consensus.cc:513] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.703176 5168 leader_election.cc:290] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 election: Requested vote from peers e6e5e1473a414d25a49e3c221025b2ed (127.0.104.3:37583), 70442fbead15433dbde12a2dcbb850f0 (127.0.104.2:42203)
I20250112 20:51:52.703887 5099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea40a8ab6594cbeb6681d9aca4138a3" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6e5e1473a414d25a49e3c221025b2ed"
I20250112 20:51:52.703958 4962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea40a8ab6594cbeb6681d9aca4138a3" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70442fbead15433dbde12a2dcbb850f0"
I20250112 20:51:52.704185 4962 raft_consensus.cc:3054] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.704298 5099 raft_consensus.cc:3054] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Advancing to term 1
W20250112 20:51:52.707990 5145 tablet.cc:2367] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:52.708439 4962 raft_consensus.cc:2463] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 70442fbead15433dbde12a2dcbb850f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 1.
I20250112 20:51:52.708962 4755 leader_election.cc:304] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70442fbead15433dbde12a2dcbb850f0, 920b0b46e8754068ab10e3cb31f8df0d; no voters:
I20250112 20:51:52.709391 5168 raft_consensus.cc:2798] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.710563 5099 raft_consensus.cc:2463] T 4ea40a8ab6594cbeb6681d9aca4138a3 P e6e5e1473a414d25a49e3c221025b2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 1.
I20250112 20:51:52.711566 5168 raft_consensus.cc:695] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [term 1 LEADER]: Becoming Leader. State: Replica: 920b0b46e8754068ab10e3cb31f8df0d, State: Running, Role: LEADER
I20250112 20:51:52.712311 5168 consensus_queue.cc:237] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } }
I20250112 20:51:52.717736 4669 catalog_manager.cc:5526] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:52.757943 5009 tablet.cc:2367] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:52.782256 4873 tablet.cc:2367] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:52.881096 5167 raft_consensus.cc:491] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:52.881304 5167 raft_consensus.cc:513] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.883198 5167 leader_election.cc:290] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1:41759), 70442fbead15433dbde12a2dcbb850f0 (127.0.104.2:42203)
I20250112 20:51:52.890074 4827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e02a0f9316945c792d84def95b67283" candidate_uuid: "e6e5e1473a414d25a49e3c221025b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920b0b46e8754068ab10e3cb31f8df0d" is_pre_election: true
I20250112 20:51:52.890290 4962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e02a0f9316945c792d84def95b67283" candidate_uuid: "e6e5e1473a414d25a49e3c221025b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70442fbead15433dbde12a2dcbb850f0" is_pre_election: true
I20250112 20:51:52.890549 4962 raft_consensus.cc:2463] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6e5e1473a414d25a49e3c221025b2ed in term 0.
I20250112 20:51:52.890571 4827 raft_consensus.cc:2463] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6e5e1473a414d25a49e3c221025b2ed in term 0.
I20250112 20:51:52.891024 5027 leader_election.cc:304] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70442fbead15433dbde12a2dcbb850f0, e6e5e1473a414d25a49e3c221025b2ed; no voters:
I20250112 20:51:52.891383 5167 raft_consensus.cc:2798] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:52.891527 5167 raft_consensus.cc:491] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:52.891640 5167 raft_consensus.cc:3054] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.893774 5167 raft_consensus.cc:513] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.894312 5167 leader_election.cc:290] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [CANDIDATE]: Term 1 election: Requested vote from peers 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1:41759), 70442fbead15433dbde12a2dcbb850f0 (127.0.104.2:42203)
I20250112 20:51:52.894737 4827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e02a0f9316945c792d84def95b67283" candidate_uuid: "e6e5e1473a414d25a49e3c221025b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "920b0b46e8754068ab10e3cb31f8df0d"
I20250112 20:51:52.894757 4962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e02a0f9316945c792d84def95b67283" candidate_uuid: "e6e5e1473a414d25a49e3c221025b2ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70442fbead15433dbde12a2dcbb850f0"
I20250112 20:51:52.894937 4827 raft_consensus.cc:3054] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.894955 4962 raft_consensus.cc:3054] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:52.897629 4962 raft_consensus.cc:2463] T 2e02a0f9316945c792d84def95b67283 P 70442fbead15433dbde12a2dcbb850f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6e5e1473a414d25a49e3c221025b2ed in term 1.
I20250112 20:51:52.897636 4827 raft_consensus.cc:2463] T 2e02a0f9316945c792d84def95b67283 P 920b0b46e8754068ab10e3cb31f8df0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6e5e1473a414d25a49e3c221025b2ed in term 1.
I20250112 20:51:52.897980 5027 leader_election.cc:304] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70442fbead15433dbde12a2dcbb850f0, e6e5e1473a414d25a49e3c221025b2ed; no voters:
I20250112 20:51:52.898296 5167 raft_consensus.cc:2798] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:52.898748 5167 raft_consensus.cc:695] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [term 1 LEADER]: Becoming Leader. State: Replica: e6e5e1473a414d25a49e3c221025b2ed, State: Running, Role: LEADER
I20250112 20:51:52.899271 5167 consensus_queue.cc:237] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:52.904042 4669 catalog_manager.cc:5526] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed reported cstate change: term changed from 0 to 1, leader changed from <none> to e6e5e1473a414d25a49e3c221025b2ed (127.0.104.3). New cstate: current_term: 1 leader_uuid: "e6e5e1473a414d25a49e3c221025b2ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:53.042044 4669 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53972:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:53.043026 4669 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:53.049196 4941 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf4c381f0c344eaac592e5e050e026c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46a460a8d0694748946a413de7497b0e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:53.049039 4806 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf4c381f0c344eaac592e5e050e026c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46a460a8d0694748946a413de7497b0e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:53.049998 4806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf4c381f0c344eaac592e5e050e026c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.049998 4941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf4c381f0c344eaac592e5e050e026c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.049638 5078 tablet_service.cc:1467] Processing CreateTablet for tablet 4cf4c381f0c344eaac592e5e050e026c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=46a460a8d0694748946a413de7497b0e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:53.050400 5078 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf4c381f0c344eaac592e5e050e026c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:53.055783 5163 tablet_bootstrap.cc:492] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: Bootstrap starting.
I20250112 20:51:53.057125 5162 tablet_bootstrap.cc:492] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: Bootstrap starting.
I20250112 20:51:53.057325 5163 tablet_bootstrap.cc:654] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.057382 5161 tablet_bootstrap.cc:492] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: Bootstrap starting.
I20250112 20:51:53.058858 5161 tablet_bootstrap.cc:654] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.058923 5162 tablet_bootstrap.cc:654] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:53.059654 5163 tablet_bootstrap.cc:492] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: No bootstrap required, opened a new log
I20250112 20:51:53.059895 5163 ts_tablet_manager.cc:1397] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:53.060477 5161 tablet_bootstrap.cc:492] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: No bootstrap required, opened a new log
I20250112 20:51:53.060657 5161 ts_tablet_manager.cc:1397] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:51:53.061086 5163 raft_consensus.cc:357] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.061249 5161 raft_consensus.cc:357] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.061388 5163 raft_consensus.cc:383] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.061444 5161 raft_consensus.cc:383] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.061517 5161 raft_consensus.cc:738] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70442fbead15433dbde12a2dcbb850f0, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.061538 5163 raft_consensus.cc:738] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e5e1473a414d25a49e3c221025b2ed, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.061769 5161 consensus_queue.cc:260] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.062269 5161 ts_tablet_manager.cc:1428] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:53.062222 5169 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.062609 5169 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:53.061942 5163 consensus_queue.cc:260] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.062978 5167 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.063244 5167 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:53.063650 5163 ts_tablet_manager.cc:1428] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:51:53.064386 5162 tablet_bootstrap.cc:492] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: No bootstrap required, opened a new log
I20250112 20:51:53.064581 5162 ts_tablet_manager.cc:1397] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250112 20:51:53.065343 5162 raft_consensus.cc:357] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.065565 5162 raft_consensus.cc:383] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:53.065652 5162 raft_consensus.cc:738] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 920b0b46e8754068ab10e3cb31f8df0d, State: Initialized, Role: FOLLOWER
I20250112 20:51:53.065927 5162 consensus_queue.cc:260] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.066478 5162 ts_tablet_manager.cc:1428] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:53.066392 5168 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.067097 5168 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:53.167909 5168 consensus_queue.cc:1035] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.176285 5168 consensus_queue.cc:1035] T 4ea40a8ab6594cbeb6681d9aca4138a3 P 920b0b46e8754068ab10e3cb31f8df0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.288435 5175 raft_consensus.cc:491] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:53.288657 5175 raft_consensus.cc:513] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.289377 5175 leader_election.cc:290] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70442fbead15433dbde12a2dcbb850f0 (127.0.104.2:42203), e6e5e1473a414d25a49e3c221025b2ed (127.0.104.3:37583)
I20250112 20:51:53.289819 4962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf4c381f0c344eaac592e5e050e026c" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70442fbead15433dbde12a2dcbb850f0" is_pre_election: true
I20250112 20:51:53.289914 5099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf4c381f0c344eaac592e5e050e026c" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6e5e1473a414d25a49e3c221025b2ed" is_pre_election: true
I20250112 20:51:53.290058 4962 raft_consensus.cc:2463] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 0.
I20250112 20:51:53.290138 5099 raft_consensus.cc:2463] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 0.
I20250112 20:51:53.290472 4755 leader_election.cc:304] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70442fbead15433dbde12a2dcbb850f0, 920b0b46e8754068ab10e3cb31f8df0d; no voters:
I20250112 20:51:53.290745 5175 raft_consensus.cc:2798] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:53.290849 5175 raft_consensus.cc:491] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:53.290939 5175 raft_consensus.cc:3054] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.292810 5175 raft_consensus.cc:513] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.293306 5175 leader_election.cc:290] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 election: Requested vote from peers 70442fbead15433dbde12a2dcbb850f0 (127.0.104.2:42203), e6e5e1473a414d25a49e3c221025b2ed (127.0.104.3:37583)
I20250112 20:51:53.293776 5099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf4c381f0c344eaac592e5e050e026c" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6e5e1473a414d25a49e3c221025b2ed"
I20250112 20:51:53.293960 5099 raft_consensus.cc:3054] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.293900 4962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cf4c381f0c344eaac592e5e050e026c" candidate_uuid: "920b0b46e8754068ab10e3cb31f8df0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70442fbead15433dbde12a2dcbb850f0"
I20250112 20:51:53.294111 4962 raft_consensus.cc:3054] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:53.295764 5099 raft_consensus.cc:2463] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 1.
I20250112 20:51:53.295904 4962 raft_consensus.cc:2463] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 920b0b46e8754068ab10e3cb31f8df0d in term 1.
I20250112 20:51:53.296162 4753 leader_election.cc:304] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 920b0b46e8754068ab10e3cb31f8df0d, e6e5e1473a414d25a49e3c221025b2ed; no voters:
I20250112 20:51:53.296443 5175 raft_consensus.cc:2798] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:53.296598 5175 raft_consensus.cc:695] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [term 1 LEADER]: Becoming Leader. State: Replica: 920b0b46e8754068ab10e3cb31f8df0d, State: Running, Role: LEADER
I20250112 20:51:53.296896 5175 consensus_queue.cc:237] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } }
I20250112 20:51:53.298152 5168 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: TxnStatusTablet state changed. Reason: New leader 920b0b46e8754068ab10e3cb31f8df0d. Latest consensus state: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.298338 5168 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:53.298892 5199 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:53.299103 5199 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:53.299633 4668 catalog_manager.cc:5526] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:53.364734 4962 raft_consensus.cc:1270] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0 [term 1 FOLLOWER]: Refusing update from remote peer 920b0b46e8754068ab10e3cb31f8df0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:53.364799 5099 raft_consensus.cc:1270] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed [term 1 FOLLOWER]: Refusing update from remote peer 920b0b46e8754068ab10e3cb31f8df0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:53.365295 5175 consensus_queue.cc:1035] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.365743 5168 consensus_queue.cc:1035] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.369575 5178 consensus_queue.cc:1035] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.374155 5167 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: TxnStatusTablet state changed. Reason: New leader 920b0b46e8754068ab10e3cb31f8df0d. Latest consensus state: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.374430 5167 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:53.376194 5192 mvcc.cc:204] Tried to move back new op lower bound from 7113585104335634432 to 7113585104065953792. Current Snapshot: MvccSnapshot[applied={T|T < 7113585104335634432}]
I20250112 20:51:53.376907 5197 mvcc.cc:204] Tried to move back new op lower bound from 7113585104335634432 to 7113585104065953792. Current Snapshot: MvccSnapshot[applied={T|T < 7113585104335634432}]
I20250112 20:51:53.377800 5167 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.378013 5167 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P e6e5e1473a414d25a49e3c221025b2ed: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:53.384343 5169 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: TxnStatusTablet state changed. Reason: New leader 920b0b46e8754068ab10e3cb31f8df0d. Latest consensus state: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.384717 5169 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:53.388911 5168 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.389145 5168 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:53.391820 5178 consensus_queue.cc:1035] T 2e02a0f9316945c792d84def95b67283 P e6e5e1473a414d25a49e3c221025b2ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:53.397284 5168 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.400179 5168 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P 920b0b46e8754068ab10e3cb31f8df0d: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:53.407002 5169 tablet_replica.cc:440] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "920b0b46e8754068ab10e3cb31f8df0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "920b0b46e8754068ab10e3cb31f8df0d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41759 } } peers { permanent_uuid: "70442fbead15433dbde12a2dcbb850f0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42203 } } peers { permanent_uuid: "e6e5e1473a414d25a49e3c221025b2ed" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37583 } } }
I20250112 20:51:53.407315 5169 tablet_replica.cc:443] T 4cf4c381f0c344eaac592e5e050e026c P 70442fbead15433dbde12a2dcbb850f0: This TxnStatusTablet replica's current role is: FOLLOWER
W20250112 20:51:53.426198 5212 tablet_replica.cc:1307] Invalid argument: Failed to write to server: 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1:41759): transaction ID 100 not found, current highest txn ID: 0: cancelling pending write operations
W20250112 20:51:53.426452 5212 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48812: Invalid argument: Failed to write to server: 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1:41759): transaction ID 100 not found, current highest txn ID: 0
W20250112 20:51:53.427439 5146 batcher.cc:441] Invalid argument: Failed to write batch of 1 ops to tablet 2e02a0f9316945c792d84def95b67283 after 1 attempt(s): Failed to write to server: e6e5e1473a414d25a49e3c221025b2ed (127.0.104.3:37583): Failed to write to server: 920b0b46e8754068ab10e3cb31f8df0d (127.0.104.1:41759): transaction ID 100 not found, current highest txn ID: 0
I20250112 20:51:53.427999 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 4738
I20250112 20:51:53.428920 4867 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:53.548863 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 4738
I20250112 20:51:53.568388 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 4874
I20250112 20:51:53.569435 5003 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:53.659979 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 4874
I20250112 20:51:53.677454 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 5010
I20250112 20:51:53.678524 5139 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:53.771688 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 5010
I20250112 20:51:53.789171 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 4638
I20250112 20:51:53.790247 4709 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:53.899277 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 4638
2025-01-12T20:51:53Z chronyd exiting
[ OK ] TxnWriteOpsITest.WriteOpForNonExistentTxn (3401 ms)
[ RUN ] TxnWriteOpsITest.TxnWriteAfterCommit
2025-01-12T20:51:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-12T20:51:53Z Disabled control of system clock
I20250112 20:51:53.969947 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35703
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35703
--txn_manager_enabled=true with env {}
W20250112 20:51:54.293260 5222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:54.293565 5222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:54.293718 5222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:54.301673 5222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250112 20:51:54.301796 5222 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20250112 20:51:54.301841 5222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:54.301927 5222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250112 20:51:54.301992 5222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250112 20:51:54.311657 5222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35703
--txn_manager_enabled=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:54.313526 5222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:54.315208 5222 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:54.325150 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:54.325196 5227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:54.325222 5228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:54.326062 5222 server_base.cc:1034] running on GCE node
I20250112 20:51:54.326817 5222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:54.328265 5222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:54.329463 5222 hybrid_clock.cc:648] HybridClock initialized: now 1736715114329417 us; error 34 us; skew 500 ppm
I20250112 20:51:54.329830 5222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:54.332075 5222 webserver.cc:458] Webserver started at http://127.0.104.62:35063/ using document root <none> and password file <none>
I20250112 20:51:54.332886 5222 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:54.332995 5222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:54.333334 5222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:54.335902 5222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data/instance:
uuid: "e568818e668f4b56ae106495916bf446"
format_stamp: "Formatted at 2025-01-12 20:51:54 on dist-test-slave-3gql"
I20250112 20:51:54.336683 5222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/wal/instance:
uuid: "e568818e668f4b56ae106495916bf446"
format_stamp: "Formatted at 2025-01-12 20:51:54 on dist-test-slave-3gql"
I20250112 20:51:54.341737 5222 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20250112 20:51:54.344996 5236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:54.346333 5222 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250112 20:51:54.346503 5222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/wal
uuid: "e568818e668f4b56ae106495916bf446"
format_stamp: "Formatted at 2025-01-12 20:51:54 on dist-test-slave-3gql"
I20250112 20:51:54.346725 5222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:54.359987 5222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:54.360899 5222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:54.361176 5222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:54.381970 5222 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35703
I20250112 20:51:54.382004 5297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35703 every 8 connection(s)
I20250112 20:51:54.383395 5222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/data/info.pb
I20250112 20:51:54.387277 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:54.388700 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 5222
I20250112 20:51:54.388901 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/master-0/wal/instance
I20250112 20:51:54.397537 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446: Bootstrap starting.
I20250112 20:51:54.401039 5298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:54.402253 5298 log.cc:826] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:54.404930 5298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446: No bootstrap required, opened a new log
I20250112 20:51:54.411197 5298 raft_consensus.cc:357] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e568818e668f4b56ae106495916bf446" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35703 } }
I20250112 20:51:54.411540 5298 raft_consensus.cc:383] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:54.411621 5298 raft_consensus.cc:738] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e568818e668f4b56ae106495916bf446, State: Initialized, Role: FOLLOWER
I20250112 20:51:54.412156 5298 consensus_queue.cc:260] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e568818e668f4b56ae106495916bf446" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35703 } }
I20250112 20:51:54.412338 5298 raft_consensus.cc:397] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:51:54.412429 5298 raft_consensus.cc:491] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:51:54.412577 5298 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:54.414688 5298 raft_consensus.cc:513] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e568818e668f4b56ae106495916bf446" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35703 } }
I20250112 20:51:54.415210 5298 leader_election.cc:304] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e568818e668f4b56ae106495916bf446; no voters:
I20250112 20:51:54.415843 5298 leader_election.cc:290] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:51:54.415970 5303 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:54.416630 5303 raft_consensus.cc:695] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [term 1 LEADER]: Becoming Leader. State: Replica: e568818e668f4b56ae106495916bf446, State: Running, Role: LEADER
I20250112 20:51:54.417217 5303 consensus_queue.cc:237] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e568818e668f4b56ae106495916bf446" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35703 } }
I20250112 20:51:54.417868 5298 sys_catalog.cc:564] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:51:54.419963 5304 sys_catalog.cc:455] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e568818e668f4b56ae106495916bf446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e568818e668f4b56ae106495916bf446" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35703 } } }
I20250112 20:51:54.420228 5304 sys_catalog.cc:458] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:54.420902 5305 sys_catalog.cc:455] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e568818e668f4b56ae106495916bf446. Latest consensus state: current_term: 1 leader_uuid: "e568818e668f4b56ae106495916bf446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e568818e668f4b56ae106495916bf446" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35703 } } }
I20250112 20:51:54.421247 5305 sys_catalog.cc:458] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446 [sys.catalog]: This master's current role is: LEADER
I20250112 20:51:54.426620 5309 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:51:54.432013 5309 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:51:54.440868 5309 catalog_manager.cc:1348] Generated new cluster ID: cf197cc9a0d74082b166c1bcaedc2f43
I20250112 20:51:54.441000 5309 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:51:54.456354 5309 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:51:54.457635 5309 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:51:54.465443 5309 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e568818e668f4b56ae106495916bf446: Generated new TSK 0
I20250112 20:51:54.466317 5309 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:51:54.477543 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35703
--builtin_ntp_servers=127.0.104.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:54.831139 5322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:54.831437 5322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:54.831569 5322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:54.839382 5322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:54.839530 5322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:54.839682 5322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:54.849045 5322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:54.850577 5322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:54.852300 5322 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:54.863927 5330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:54.863898 5328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:54.863974 5327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:54.864828 5322 server_base.cc:1034] running on GCE node
I20250112 20:51:54.865410 5322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:54.866315 5322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:54.867525 5322 hybrid_clock.cc:648] HybridClock initialized: now 1736715114867476 us; error 49 us; skew 500 ppm
I20250112 20:51:54.867828 5322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:54.869992 5322 webserver.cc:458] Webserver started at http://127.0.104.1:36663/ using document root <none> and password file <none>
I20250112 20:51:54.870730 5322 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:54.870831 5322 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:54.871148 5322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:54.873688 5322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/instance:
uuid: "ea410e5f892a427b893e5f4729675c1a"
format_stamp: "Formatted at 2025-01-12 20:51:54 on dist-test-slave-3gql"
I20250112 20:51:54.874430 5322 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal/instance:
uuid: "ea410e5f892a427b893e5f4729675c1a"
format_stamp: "Formatted at 2025-01-12 20:51:54 on dist-test-slave-3gql"
I20250112 20:51:54.879518 5322 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250112 20:51:54.882865 5336 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:54.884187 5322 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:54.884382 5322 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "ea410e5f892a427b893e5f4729675c1a"
format_stamp: "Formatted at 2025-01-12 20:51:54 on dist-test-slave-3gql"
I20250112 20:51:54.884639 5322 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:54.903940 5322 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:54.904819 5322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:54.905092 5322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:54.907919 5322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:54.908044 5322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:54.908188 5322 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:54.908248 5322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:54.956351 5322 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34889
I20250112 20:51:54.956401 5456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34889 every 8 connection(s)
I20250112 20:51:54.957712 5322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:54.959882 5457 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35703
I20250112 20:51:54.960114 5457 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:54.960968 5457 heartbeater.cc:510] Master 127.0.104.62:35703 requested a full tablet report, sending...
I20250112 20:51:54.961196 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 5322
I20250112 20:51:54.961416 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal/instance
I20250112 20:51:54.963300 5253 ts_manager.cc:194] Registered new tserver with Master: ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889)
I20250112 20:51:54.965147 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35703
--builtin_ntp_servers=127.0.104.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
I20250112 20:51:54.965260 5253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:38633
W20250112 20:51:55.297086 5459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:55.297410 5459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:55.297552 5459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:55.305138 5459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:55.305330 5459 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:55.305482 5459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:55.314538 5459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:55.316241 5459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:55.317975 5459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:55.329280 5467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:55.329325 5465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:55.329283 5464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:55.330287 5459 server_base.cc:1034] running on GCE node
I20250112 20:51:55.330835 5459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:55.331727 5459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:55.332937 5459 hybrid_clock.cc:648] HybridClock initialized: now 1736715115332888 us; error 36 us; skew 500 ppm
I20250112 20:51:55.333240 5459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:55.335486 5459 webserver.cc:458] Webserver started at http://127.0.104.2:33525/ using document root <none> and password file <none>
I20250112 20:51:55.336241 5459 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:55.336350 5459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:55.336697 5459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:55.339344 5459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/instance:
uuid: "1cdfc7648c274dec813577a70a67e12d"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:55.340101 5459 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal/instance:
uuid: "1cdfc7648c274dec813577a70a67e12d"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:55.345422 5459 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20250112 20:51:55.348750 5473 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:55.350087 5459 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250112 20:51:55.350246 5459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "1cdfc7648c274dec813577a70a67e12d"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:55.350438 5459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:55.362960 5459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:55.363773 5459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:55.364054 5459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:55.366876 5459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:55.367023 5459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:55.367211 5459 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:55.367285 5459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:55.415357 5459 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38157
I20250112 20:51:55.415408 5592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38157 every 8 connection(s)
I20250112 20:51:55.416649 5459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:55.418776 5593 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35703
I20250112 20:51:55.419096 5593 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:55.419898 5593 heartbeater.cc:510] Master 127.0.104.62:35703 requested a full tablet report, sending...
I20250112 20:51:55.421244 5253 ts_manager.cc:194] Registered new tserver with Master: 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157)
I20250112 20:51:55.422183 5253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:55965
I20250112 20:51:55.425906 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 5459
I20250112 20:51:55.426056 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal/instance
I20250112 20:51:55.428872 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35703
--builtin_ntp_servers=127.0.104.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:55.753997 5595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:55.754321 5595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:55.754470 5595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:55.762262 5595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:55.762470 5595 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:55.762614 5595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:55.771941 5595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:55.773516 5595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:55.775141 5595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:55.786672 5601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:55.787015 5595 server_base.cc:1034] running on GCE node
W20250112 20:51:55.786653 5603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:55.786656 5600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:55.787902 5595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:55.788800 5595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:55.790053 5595 hybrid_clock.cc:648] HybridClock initialized: now 1736715115789994 us; error 49 us; skew 500 ppm
I20250112 20:51:55.790338 5595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:55.792645 5595 webserver.cc:458] Webserver started at http://127.0.104.3:44707/ using document root <none> and password file <none>
I20250112 20:51:55.793403 5595 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:55.793493 5595 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:55.793780 5595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:51:55.796272 5595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/instance:
uuid: "737fa552528642308bfd0388e079ad88"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:55.797096 5595 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal/instance:
uuid: "737fa552528642308bfd0388e079ad88"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:55.802121 5595 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250112 20:51:55.805337 5609 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:55.806595 5595 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250112 20:51:55.806751 5595 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "737fa552528642308bfd0388e079ad88"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:55.806941 5595 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:55.820778 5595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:55.821573 5595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:55.821833 5595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:55.824647 5595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:51:55.824734 5595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:55.824918 5595 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:51:55.825022 5595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:55.868360 5595 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46001
I20250112 20:51:55.868391 5728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46001 every 8 connection(s)
I20250112 20:51:55.869665 5595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:55.871728 5729 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35703
I20250112 20:51:55.871933 5729 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:55.872656 5729 heartbeater.cc:510] Master 127.0.104.62:35703 requested a full tablet report, sending...
I20250112 20:51:55.873876 5253 ts_manager.cc:194] Registered new tserver with Master: 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:55.874606 5253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:40005
I20250112 20:51:55.878506 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 5595
I20250112 20:51:55.878654 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal/instance
I20250112 20:51:55.881564 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250112 20:51:55.896660 5253 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55824:
name: "txn_write_ops_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:51:55.902685 5253 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_write_ops_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:55.946044 5663 tablet_service.cc:1467] Processing CreateTablet for tablet de9ddce96cce47f7aaeb0dfa4db65ad3 (DEFAULT_TABLE table=txn_write_ops_test [id=e95a4605d65f4dcba7fd0491cf74f33d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.945919 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8f9ac7de6d4211bf4ff89b7ca5acbf (DEFAULT_TABLE table=txn_write_ops_test [id=e95a4605d65f4dcba7fd0491cf74f33d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.947345 5392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8f9ac7de6d4211bf4ff89b7ca5acbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.946995 5526 tablet_service.cc:1467] Processing CreateTablet for tablet de9ddce96cce47f7aaeb0dfa4db65ad3 (DEFAULT_TABLE table=txn_write_ops_test [id=e95a4605d65f4dcba7fd0491cf74f33d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.947135 5527 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8f9ac7de6d4211bf4ff89b7ca5acbf (DEFAULT_TABLE table=txn_write_ops_test [id=e95a4605d65f4dcba7fd0491cf74f33d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.947610 5391 tablet_service.cc:1467] Processing CreateTablet for tablet de9ddce96cce47f7aaeb0dfa4db65ad3 (DEFAULT_TABLE table=txn_write_ops_test [id=e95a4605d65f4dcba7fd0491cf74f33d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.945881 5664 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8f9ac7de6d4211bf4ff89b7ca5acbf (DEFAULT_TABLE table=txn_write_ops_test [id=e95a4605d65f4dcba7fd0491cf74f33d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:51:55.948177 5663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9ddce96cce47f7aaeb0dfa4db65ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.948195 5527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8f9ac7de6d4211bf4ff89b7ca5acbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.948403 5391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9ddce96cce47f7aaeb0dfa4db65ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.951704 5664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8f9ac7de6d4211bf4ff89b7ca5acbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.951890 5526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de9ddce96cce47f7aaeb0dfa4db65ad3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:55.964952 5747 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Bootstrap starting.
I20250112 20:51:55.968587 5457 heartbeater.cc:502] Master 127.0.104.62:35703 was elected leader, sending a full tablet report...
I20250112 20:51:55.969729 5747 tablet_bootstrap.cc:654] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.971462 5748 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Bootstrap starting.
I20250112 20:51:55.971423 5747 log.cc:826] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:55.975625 5748 tablet_bootstrap.cc:654] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.977353 5748 log.cc:826] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:55.987191 5746 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Bootstrap starting.
I20250112 20:51:55.987700 5748 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: No bootstrap required, opened a new log
I20250112 20:51:55.988009 5748 ts_tablet_manager.cc:1397] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20250112 20:51:55.989075 5747 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: No bootstrap required, opened a new log
I20250112 20:51:55.989324 5747 ts_tablet_manager.cc:1397] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.010s
I20250112 20:51:55.990965 5746 tablet_bootstrap.cc:654] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:55.992280 5746 log.cc:826] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:55.995086 5746 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: No bootstrap required, opened a new log
I20250112 20:51:55.995358 5746 ts_tablet_manager.cc:1397] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250112 20:51:55.995640 5747 raft_consensus.cc:357] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:55.995981 5747 raft_consensus.cc:383] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:55.996112 5747 raft_consensus.cc:738] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737fa552528642308bfd0388e079ad88, State: Initialized, Role: FOLLOWER
I20250112 20:51:55.996798 5747 consensus_queue.cc:260] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:55.996757 5748 raft_consensus.cc:357] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:55.997211 5748 raft_consensus.cc:383] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:55.997355 5748 raft_consensus.cc:738] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Initialized, Role: FOLLOWER
I20250112 20:51:55.998104 5748 consensus_queue.cc:260] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:56.001142 5748 ts_tablet_manager.cc:1428] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20250112 20:51:56.001652 5748 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Bootstrap starting.
I20250112 20:51:56.003548 5748 tablet_bootstrap.cc:654] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:56.004083 5729 heartbeater.cc:502] Master 127.0.104.62:35703 was elected leader, sending a full tablet report...
I20250112 20:51:56.004412 5746 raft_consensus.cc:357] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:56.004933 5746 raft_consensus.cc:383] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:56.004979 5747 ts_tablet_manager.cc:1428] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20250112 20:51:56.005075 5746 raft_consensus.cc:738] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdfc7648c274dec813577a70a67e12d, State: Initialized, Role: FOLLOWER
I20250112 20:51:56.005918 5746 consensus_queue.cc:260] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:56.006446 5747 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Bootstrap starting.
I20250112 20:51:56.008997 5747 tablet_bootstrap.cc:654] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:56.010775 5593 heartbeater.cc:502] Master 127.0.104.62:35703 was elected leader, sending a full tablet report...
I20250112 20:51:56.011109 5746 ts_tablet_manager.cc:1428] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20250112 20:51:56.011510 5746 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Bootstrap starting.
I20250112 20:51:56.013243 5746 tablet_bootstrap.cc:654] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:56.015554 5748 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: No bootstrap required, opened a new log
I20250112 20:51:56.015796 5748 ts_tablet_manager.cc:1397] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20250112 20:51:56.016759 5747 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: No bootstrap required, opened a new log
I20250112 20:51:56.016978 5747 ts_tablet_manager.cc:1397] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250112 20:51:56.016880 5748 raft_consensus.cc:357] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.017102 5748 raft_consensus.cc:383] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:56.017227 5748 raft_consensus.cc:738] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Initialized, Role: FOLLOWER
I20250112 20:51:56.017537 5748 consensus_queue.cc:260] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.017978 5747 raft_consensus.cc:357] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.018113 5748 ts_tablet_manager.cc:1428] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:56.018252 5747 raft_consensus.cc:383] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:56.018353 5747 raft_consensus.cc:738] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737fa552528642308bfd0388e079ad88, State: Initialized, Role: FOLLOWER
I20250112 20:51:56.018625 5747 consensus_queue.cc:260] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.019088 5746 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: No bootstrap required, opened a new log
I20250112 20:51:56.019291 5746 ts_tablet_manager.cc:1397] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20250112 20:51:56.019312 5747 ts_tablet_manager.cc:1428] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:56.020037 5746 raft_consensus.cc:357] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.020244 5746 raft_consensus.cc:383] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:56.020378 5746 raft_consensus.cc:738] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdfc7648c274dec813577a70a67e12d, State: Initialized, Role: FOLLOWER
I20250112 20:51:56.020735 5746 consensus_queue.cc:260] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.021178 5746 ts_tablet_manager.cc:1428] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250112 20:51:56.082075 5752 raft_consensus.cc:491] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:56.082371 5752 raft_consensus.cc:513] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:56.084041 5752 leader_election.cc:290] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889)
I20250112 20:51:56.090435 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d" is_pre_election: true
I20250112 20:51:56.090934 5752 raft_consensus.cc:491] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:56.090947 5548 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 0.
I20250112 20:51:56.090948 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea410e5f892a427b893e5f4729675c1a" is_pre_election: true
I20250112 20:51:56.091113 5752 raft_consensus.cc:513] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.091413 5412 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 0.
I20250112 20:51:56.091688 5752 leader_election.cc:290] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889), 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157)
I20250112 20:51:56.091621 5612 leader_election.cc:304] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d, 737fa552528642308bfd0388e079ad88; no voters:
I20250112 20:51:56.092065 5752 raft_consensus.cc:2798] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:56.092198 5752 raft_consensus.cc:491] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:56.092247 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea410e5f892a427b893e5f4729675c1a" is_pre_election: true
I20250112 20:51:56.092326 5752 raft_consensus.cc:3054] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.092438 5412 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 0.
I20250112 20:51:56.092694 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d" is_pre_election: true
I20250112 20:51:56.092801 5613 leader_election.cc:304] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737fa552528642308bfd0388e079ad88, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:56.092896 5548 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 0.
I20250112 20:51:56.093489 5759 raft_consensus.cc:2798] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:56.093578 5759 raft_consensus.cc:491] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:56.093725 5759 raft_consensus.cc:3054] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.095026 5752 raft_consensus.cc:513] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:56.095613 5752 leader_election.cc:290] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 election: Requested vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889)
I20250112 20:51:56.095808 5759 raft_consensus.cc:513] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.095991 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d"
I20250112 20:51:56.096062 5412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea410e5f892a427b893e5f4729675c1a"
I20250112 20:51:56.096189 5548 raft_consensus.cc:3054] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.096235 5412 raft_consensus.cc:3054] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.096482 5759 leader_election.cc:290] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 election: Requested vote from peers ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889), 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157)
I20250112 20:51:56.096944 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea410e5f892a427b893e5f4729675c1a"
I20250112 20:51:56.097276 5411 raft_consensus.cc:3054] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.098338 5548 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 1.
I20250112 20:51:56.098830 5612 leader_election.cc:304] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d, 737fa552528642308bfd0388e079ad88; no voters:
I20250112 20:51:56.098995 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d"
I20250112 20:51:56.099189 5548 raft_consensus.cc:3054] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.099184 5759 raft_consensus.cc:2798] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:56.099422 5759 raft_consensus.cc:695] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 1 LEADER]: Becoming Leader. State: Replica: 737fa552528642308bfd0388e079ad88, State: Running, Role: LEADER
I20250112 20:51:56.099610 5412 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 1.
I20250112 20:51:56.099973 5411 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 1.
I20250112 20:51:56.099960 5759 consensus_queue.cc:237] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:56.101403 5613 leader_election.cc:304] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737fa552528642308bfd0388e079ad88, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:56.101671 5752 raft_consensus.cc:2798] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:56.101675 5548 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 1.
I20250112 20:51:56.102205 5752 raft_consensus.cc:695] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 1 LEADER]: Becoming Leader. State: Replica: 737fa552528642308bfd0388e079ad88, State: Running, Role: LEADER
I20250112 20:51:56.102560 5752 consensus_queue.cc:237] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.105552 5253 catalog_manager.cc:5526] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 737fa552528642308bfd0388e079ad88 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:56.110476 5252 catalog_manager.cc:5526] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 737fa552528642308bfd0388e079ad88 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } health_report { overall_health: HEALTHY } } }
W20250112 20:51:56.121357 5730 tablet.cc:2367] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:56.168912 5594 tablet.cc:2367] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250112 20:51:56.210155 5458 tablet.cc:2367] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:51:56.217240 5252 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55834:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:51:56.218289 5252 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:51:56.224831 5527 tablet_service.cc:1467] Processing CreateTablet for tablet 1bbd5f1d33c746a687c1892773195f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4503bccada042bba5108b1d783ff508]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:56.224742 5392 tablet_service.cc:1467] Processing CreateTablet for tablet 1bbd5f1d33c746a687c1892773195f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4503bccada042bba5108b1d783ff508]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:56.225522 5392 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbd5f1d33c746a687c1892773195f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:56.225528 5527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbd5f1d33c746a687c1892773195f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:56.228161 5664 tablet_service.cc:1467] Processing CreateTablet for tablet 1bbd5f1d33c746a687c1892773195f1d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e4503bccada042bba5108b1d783ff508]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:51:56.229000 5664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bbd5f1d33c746a687c1892773195f1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:51:56.230777 5748 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Bootstrap starting.
I20250112 20:51:56.232468 5748 tablet_bootstrap.cc:654] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:56.234421 5748 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: No bootstrap required, opened a new log
I20250112 20:51:56.234575 5748 ts_tablet_manager.cc:1397] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:56.235116 5747 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Bootstrap starting.
I20250112 20:51:56.235417 5748 raft_consensus.cc:357] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.235622 5748 raft_consensus.cc:383] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:56.235692 5748 raft_consensus.cc:738] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Initialized, Role: FOLLOWER
I20250112 20:51:56.236279 5748 consensus_queue.cc:260] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.236999 5747 tablet_bootstrap.cc:654] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:56.236778 5753 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.237119 5753 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:56.237561 5748 ts_tablet_manager.cc:1428] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:56.238485 5746 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Bootstrap starting.
I20250112 20:51:56.239423 5747 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: No bootstrap required, opened a new log
I20250112 20:51:56.239586 5747 ts_tablet_manager.cc:1397] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:51:56.240206 5746 tablet_bootstrap.cc:654] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Neither blocks nor log segments found. Creating new log.
I20250112 20:51:56.240399 5747 raft_consensus.cc:357] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.240597 5747 raft_consensus.cc:383] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:56.240701 5747 raft_consensus.cc:738] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737fa552528642308bfd0388e079ad88, State: Initialized, Role: FOLLOWER
I20250112 20:51:56.240973 5747 consensus_queue.cc:260] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.241643 5747 ts_tablet_manager.cc:1428] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:51:56.241554 5752 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.241864 5752 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:56.242183 5746 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: No bootstrap required, opened a new log
I20250112 20:51:56.242334 5746 ts_tablet_manager.cc:1397] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:51:56.243146 5746 raft_consensus.cc:357] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.243399 5746 raft_consensus.cc:383] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:51:56.243482 5746 raft_consensus.cc:738] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdfc7648c274dec813577a70a67e12d, State: Initialized, Role: FOLLOWER
I20250112 20:51:56.243795 5746 consensus_queue.cc:260] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.244289 5746 ts_tablet_manager.cc:1428] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:56.244212 5755 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.244571 5755 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:56.383319 5752 raft_consensus.cc:491] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:56.383519 5752 raft_consensus.cc:513] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.384284 5752 leader_election.cc:290] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889), 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157)
I20250112 20:51:56.384769 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea410e5f892a427b893e5f4729675c1a" is_pre_election: true
I20250112 20:51:56.384949 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d" is_pre_election: true
I20250112 20:51:56.385030 5411 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 0.
I20250112 20:51:56.385169 5548 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 0.
I20250112 20:51:56.385502 5613 leader_election.cc:304] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737fa552528642308bfd0388e079ad88, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:56.385879 5752 raft_consensus.cc:2798] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:51:56.386025 5752 raft_consensus.cc:491] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:51:56.386134 5752 raft_consensus.cc:3054] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.389042 5752 raft_consensus.cc:513] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.389814 5752 leader_election.cc:290] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 election: Requested vote from peers ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889), 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157)
I20250112 20:51:56.390264 5548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d"
I20250112 20:51:56.390431 5548 raft_consensus.cc:3054] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.392284 5548 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 1.
I20250112 20:51:56.392743 5612 leader_election.cc:304] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d, 737fa552528642308bfd0388e079ad88; no voters:
I20250112 20:51:56.392753 5411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "737fa552528642308bfd0388e079ad88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea410e5f892a427b893e5f4729675c1a"
I20250112 20:51:56.393088 5411 raft_consensus.cc:3054] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:51:56.393128 5752 raft_consensus.cc:2798] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:51:56.393260 5752 raft_consensus.cc:695] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 1 LEADER]: Becoming Leader. State: Replica: 737fa552528642308bfd0388e079ad88, State: Running, Role: LEADER
I20250112 20:51:56.393558 5752 consensus_queue.cc:237] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:56.395032 5759 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: TxnStatusTablet state changed. Reason: New leader 737fa552528642308bfd0388e079ad88. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.395255 5759 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:56.395779 5774 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:56.395895 5411 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737fa552528642308bfd0388e079ad88 in term 1.
I20250112 20:51:56.396099 5774 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:56.397434 5252 catalog_manager.cc:5526] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 737fa552528642308bfd0388e079ad88 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:56.439591 5411 raft_consensus.cc:1270] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Refusing update from remote peer 737fa552528642308bfd0388e079ad88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:56.439635 5548 raft_consensus.cc:1270] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Refusing update from remote peer 737fa552528642308bfd0388e079ad88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:56.440667 5752 consensus_queue.cc:1035] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:56.441320 5752 consensus_queue.cc:1035] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:56.456725 5755 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: TxnStatusTablet state changed. Reason: New leader 737fa552528642308bfd0388e079ad88. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.457026 5755 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:56.460942 5753 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: TxnStatusTablet state changed. Reason: New leader 737fa552528642308bfd0388e079ad88. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.461256 5753 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:56.461283 5777 mvcc.cc:204] Tried to move back new op lower bound from 7113585116926410752 to 7113585116750340096. Current Snapshot: MvccSnapshot[applied={T|T < 7113585116926410752}]
I20250112 20:51:56.463438 5778 mvcc.cc:204] Tried to move back new op lower bound from 7113585116926410752 to 7113585116750340096. Current Snapshot: MvccSnapshot[applied={T|T < 7113585116926410752}]
I20250112 20:51:56.464706 5759 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.464944 5759 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:56.464643 5752 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.466240 5752 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:56.478196 5755 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.478577 5755 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:56.482632 5753 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:56.482913 5753 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:56.517748 5411 raft_consensus.cc:1270] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Refusing update from remote peer 737fa552528642308bfd0388e079ad88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:56.517760 5548 raft_consensus.cc:1270] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Refusing update from remote peer 737fa552528642308bfd0388e079ad88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:51:56.518296 5760 consensus_queue.cc:1035] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:56.518829 5752 consensus_queue.cc:1035] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250112 20:51:56.574534 5787 tablet_replica.cc:1307] Illegal state: Failed to write to server: 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001): transaction ID 0 not open: COMMITTED: cancelling pending write operations
W20250112 20:51:56.574802 5787 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44016: Illegal state: Failed to write to server: 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001): transaction ID 0 not open: COMMITTED
W20250112 20:51:56.575472 5732 batcher.cc:441] Illegal state: Failed to write batch of 1 ops to tablet de9ddce96cce47f7aaeb0dfa4db65ad3 after 1 attempt(s): Failed to write to server: 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001): Failed to write to server: 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001): transaction ID 0 not open: COMMITTED
I20250112 20:51:56.593802 5759 consensus_queue.cc:1035] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:56.638339 5752 consensus_queue.cc:1035] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:51:56.658903 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 5322
I20250112 20:51:56.659952 5452 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:56.757640 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 5322
I20250112 20:51:56.776151 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:34889
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:35703
--webserver_port=36663
--webserver_interface=127.0.104.1
--builtin_ntp_servers=127.0.104.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:57.023933 5613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34889: connect: Connection refused (error 111)
W20250112 20:51:57.028301 5613 consensus_peers.cc:487] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 -> Peer ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889): Couldn't send request to peer ea410e5f892a427b893e5f4729675c1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:57.102128 5801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:57.102416 5801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:57.102559 5801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:57.109927 5801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:57.110107 5801 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:57.110225 5801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250112 20:51:57.120019 5801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:34889
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=36663
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:57.121770 5801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:57.123456 5801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:57.135334 5809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:57.135324 5808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:57.136411 5811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:57.136785 5801 server_base.cc:1034] running on GCE node
I20250112 20:51:57.137418 5801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:57.138378 5801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:57.139583 5801 hybrid_clock.cc:648] HybridClock initialized: now 1736715117139516 us; error 56 us; skew 500 ppm
I20250112 20:51:57.139906 5801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:57.142186 5801 webserver.cc:458] Webserver started at http://127.0.104.1:36663/ using document root <none> and password file <none>
I20250112 20:51:57.142925 5801 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:57.143034 5801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:57.148674 5801 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250112 20:51:57.153399 5818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:57.154806 5801 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:57.154956 5801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
uuid: "ea410e5f892a427b893e5f4729675c1a"
format_stamp: "Formatted at 2025-01-12 20:51:54 on dist-test-slave-3gql"
I20250112 20:51:57.155805 5801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:57.173872 5801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:57.174690 5801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:57.174964 5801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:57.178472 5826 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250112 20:51:57.191170 5801 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250112 20:51:57.191289 5801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.000s
I20250112 20:51:57.191421 5801 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250112 20:51:57.195052 5826 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Bootstrap starting.
I20250112 20:51:57.200393 5801 ts_tablet_manager.cc:610] Registered 3 tablets
I20250112 20:51:57.200557 5801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.009s sys 0.000s
I20250112 20:51:57.220607 5826 log.cc:826] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Log is configured to *not* fsync() on all Append() calls
W20250112 20:51:57.228724 5613 consensus_peers.cc:487] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 -> Peer ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889): Couldn't send request to peer ea410e5f892a427b893e5f4729675c1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34889: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:57.239874 5826 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:57.239885 5801 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34889
I20250112 20:51:57.239902 5940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34889 every 8 connection(s)
I20250112 20:51:57.240448 5826 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Bootstrap complete.
I20250112 20:51:57.241200 5801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-0/data/info.pb
I20250112 20:51:57.241619 5826 ts_tablet_manager.cc:1397] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Time spent bootstrapping tablet: real 0.047s user 0.025s sys 0.020s
I20250112 20:51:57.243861 5941 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35703
I20250112 20:51:57.244067 5941 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:57.244872 5941 heartbeater.cc:510] Master 127.0.104.62:35703 requested a full tablet report, sending...
I20250112 20:51:57.247442 5252 ts_manager.cc:194] Re-registered known tserver with Master: ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889)
I20250112 20:51:57.248178 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 5801
I20250112 20:51:57.248448 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 5459
I20250112 20:51:57.247705 5826 raft_consensus.cc:357] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:57.248956 5826 raft_consensus.cc:738] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Initialized, Role: FOLLOWER
I20250112 20:51:57.249586 5588 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:57.249809 5252 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:53349
I20250112 20:51:57.249701 5826 consensus_queue.cc:260] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:57.251979 5826 ts_tablet_manager.cc:1428] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20250112 20:51:57.252444 5826 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Bootstrap starting.
I20250112 20:51:57.255366 5941 heartbeater.cc:502] Master 127.0.104.62:35703 was elected leader, sending a full tablet report...
I20250112 20:51:57.265537 5826 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:57.265957 5826 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Bootstrap complete.
I20250112 20:51:57.266974 5826 ts_tablet_manager.cc:1397] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250112 20:51:57.267580 5826 raft_consensus.cc:357] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.267752 5826 raft_consensus.cc:738] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Initialized, Role: FOLLOWER
I20250112 20:51:57.267985 5826 consensus_queue.cc:260] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.268468 5826 ts_tablet_manager.cc:1428] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:57.268746 5826 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Bootstrap starting.
I20250112 20:51:57.291211 5826 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=4 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:57.291851 5826 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Bootstrap complete.
I20250112 20:51:57.293267 5826 ts_tablet_manager.cc:1397] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.012s
I20250112 20:51:57.294142 5826 raft_consensus.cc:357] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.294376 5826 raft_consensus.cc:738] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Initialized, Role: FOLLOWER
W20250112 20:51:57.294509 5613 consensus_peers.cc:487] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 -> Peer ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889): Couldn't send request to peer ea410e5f892a427b893e5f4729675c1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:57.294657 5826 consensus_queue.cc:260] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.295181 5826 ts_tablet_manager.cc:1428] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250112 20:51:57.295080 5944 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:57.295428 5944 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:57.346525 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 5459
I20250112 20:51:57.364382 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:38157
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:35703
--webserver_port=33525
--webserver_interface=127.0.104.2
--builtin_ntp_servers=127.0.104.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:57.548424 5612 consensus_peers.cc:487] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 -> Peer 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157): Couldn't send request to peer 1cdfc7648c274dec813577a70a67e12d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.2:38157: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250112 20:51:57.684496 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:57.684846 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:57.684998 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:57.692734 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:57.692929 5948 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:57.693045 5948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250112 20:51:57.702179 5948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:38157
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=33525
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:57.703789 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:57.705644 5948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:57.716681 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:57.716650 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:57.716720 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:57.717782 5948 server_base.cc:1034] running on GCE node
I20250112 20:51:57.718371 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:57.719475 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:57.720631 5948 hybrid_clock.cc:648] HybridClock initialized: now 1736715117720592 us; error 34 us; skew 500 ppm
I20250112 20:51:57.720933 5948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:57.723138 5948 webserver.cc:458] Webserver started at http://127.0.104.2:33525/ using document root <none> and password file <none>
I20250112 20:51:57.723848 5948 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:57.723974 5948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:57.729210 5948 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20250112 20:51:57.732123 5962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:57.733457 5948 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250112 20:51:57.733606 5948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
uuid: "1cdfc7648c274dec813577a70a67e12d"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:57.734382 5948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:57.758181 5948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:57.759014 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:57.759289 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:57.762745 5970 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250112 20:51:57.774659 5948 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250112 20:51:57.774794 5948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.000s
I20250112 20:51:57.774937 5948 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250112 20:51:57.778730 5970 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Bootstrap starting.
I20250112 20:51:57.783213 5948 ts_tablet_manager.cc:610] Registered 3 tablets
I20250112 20:51:57.783339 5948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
W20250112 20:51:57.788795 5612 consensus_peers.cc:487] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 -> Peer 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157): Couldn't send request to peer 1cdfc7648c274dec813577a70a67e12d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.2:38157: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250112 20:51:57.799772 5970 log.cc:826] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:57.819510 5970 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:57.820341 5970 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Bootstrap complete.
I20250112 20:51:57.821905 5970 ts_tablet_manager.cc:1397] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Time spent bootstrapping tablet: real 0.044s user 0.037s sys 0.004s
I20250112 20:51:57.828745 5970 raft_consensus.cc:357] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:57.830088 5970 raft_consensus.cc:738] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdfc7648c274dec813577a70a67e12d, State: Initialized, Role: FOLLOWER
I20250112 20:51:57.831070 5970 consensus_queue.cc:260] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:57.832790 5970 ts_tablet_manager.cc:1428] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20250112 20:51:57.833287 5970 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Bootstrap starting.
I20250112 20:51:57.839388 5948 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38157
I20250112 20:51:57.839479 6085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38157 every 8 connection(s)
I20250112 20:51:57.841531 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-1/data/info.pb
I20250112 20:51:57.844344 6087 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35703
I20250112 20:51:57.844679 6087 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:57.844717 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 5948
I20250112 20:51:57.845003 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 5595
I20250112 20:51:57.846014 6087 heartbeater.cc:510] Master 127.0.104.62:35703 requested a full tablet report, sending...
I20250112 20:51:57.846139 5724 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:57.849043 5253 ts_manager.cc:194] Re-registered known tserver with Master: 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157)
I20250112 20:51:57.851783 5253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:44065
I20250112 20:51:57.855991 5970 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:57.856415 5970 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Bootstrap complete.
I20250112 20:51:57.857396 5970 ts_tablet_manager.cc:1397] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.011s
I20250112 20:51:57.858017 5970 raft_consensus.cc:357] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.858175 5970 raft_consensus.cc:738] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdfc7648c274dec813577a70a67e12d, State: Initialized, Role: FOLLOWER
I20250112 20:51:57.858392 5970 consensus_queue.cc:260] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.858769 6087 heartbeater.cc:502] Master 127.0.104.62:35703 was elected leader, sending a full tablet report...
I20250112 20:51:57.859433 5970 ts_tablet_manager.cc:1428] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:51:57.859660 5970 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Bootstrap starting.
I20250112 20:51:57.880264 5970 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=4 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:57.880792 5970 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Bootstrap complete.
I20250112 20:51:57.881728 5970 ts_tablet_manager.cc:1397] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.012s
I20250112 20:51:57.882405 5970 raft_consensus.cc:357] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.882594 5970 raft_consensus.cc:738] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdfc7648c274dec813577a70a67e12d, State: Initialized, Role: FOLLOWER
I20250112 20:51:57.882875 5970 consensus_queue.cc:260] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:57.883445 5970 ts_tablet_manager.cc:1428] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:57.883293 6075 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:57.883649 6075 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:57.968653 6092 tablet_replica.cc:440] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:57.969022 6092 tablet_replica.cc:443] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:57.969337 5944 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: New leader 737fa552528642308bfd0388e079ad88. Latest consensus state: current_term: 1 leader_uuid: "737fa552528642308bfd0388e079ad88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:57.969542 5944 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:57.971217 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 5595
I20250112 20:51:57.993956 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskRSg17H/build/asan/bin/kudu
/tmp/dist-test-taskRSg17H/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:46001
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:35703
--webserver_port=44707
--webserver_interface=127.0.104.3
--builtin_ntp_servers=127.0.104.20:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_txn_system_client_init=true with env {}
W20250112 20:51:57.994127 6086 negotiation.cc:337] Failed RPC negotiation. Trace:
0112 20:51:57.840008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:34727 (local address 127.0.104.2:38157)
0112 20:51:57.843675 (+ 3667us) server_negotiation.cc:206] Beginning negotiation
0112 20:51:57.843685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0112 20:51:57.843723 (+ 38us) server_negotiation.cc:407] Connection header received
0112 20:51:57.843855 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0112 20:51:57.843867 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0112 20:51:57.844273 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0112 20:51:57.848752 (+ 4479us) server_negotiation.cc:226] Negotiated authn=SASL
0112 20:51:57.849161 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0112 20:51:57.851724 (+ 2563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0112 20:51:57.973419 (+121695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0112 20:51:57.973894 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0112 20:51:57.992789 (+ 18895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:34727: BlockingRecv error: recv got EOF from 127.0.104.3:34727 (error 108)
Metrics: {"server-negotiator.queue_time_us":3485,"thread_start_us":3377,"threads_started":1}
W20250112 20:51:58.318496 6094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250112 20:51:58.318845 6094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250112 20:51:58.318997 6094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250112 20:51:58.326853 6094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250112 20:51:58.327054 6094 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20250112 20:51:58.327163 6094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250112 20:51:58.336449 6094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:46001
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=44707
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:35703
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 12 Jan 2025 20:43:23 UTC on 24a791456cd2
build id 4147
ASAN enabled
I20250112 20:51:58.338047 6094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:51:58.339723 6094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:51:58.351114 6102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:58.351073 6100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:51:58.351161 6099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:51:58.351913 6094 server_base.cc:1034] running on GCE node
I20250112 20:51:58.352499 6094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250112 20:51:58.353483 6094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250112 20:51:58.354720 6094 hybrid_clock.cc:648] HybridClock initialized: now 1736715118354672 us; error 34 us; skew 500 ppm
I20250112 20:51:58.355039 6094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:51:58.357285 6094 webserver.cc:458] Webserver started at http://127.0.104.3:44707/ using document root <none> and password file <none>
I20250112 20:51:58.358103 6094 fs_manager.cc:362] Metadata directory not provided
I20250112 20:51:58.358230 6094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:51:58.364022 6094 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250112 20:51:58.366950 6108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:51:58.368177 6094 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250112 20:51:58.368323 6094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
uuid: "737fa552528642308bfd0388e079ad88"
format_stamp: "Formatted at 2025-01-12 20:51:55 on dist-test-slave-3gql"
I20250112 20:51:58.369185 6094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:51:58.412688 6094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:51:58.413558 6094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:51:58.413838 6094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:51:58.417238 6116 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250112 20:51:58.429021 6094 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250112 20:51:58.429162 6094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.001s
I20250112 20:51:58.429283 6094 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250112 20:51:58.433071 6116 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Bootstrap starting.
I20250112 20:51:58.437331 6094 ts_tablet_manager.cc:610] Registered 3 tablets
I20250112 20:51:58.437433 6094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.005s sys 0.002s
I20250112 20:51:58.453531 6116 log.cc:826] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Log is configured to *not* fsync() on all Append() calls
I20250112 20:51:58.477133 6116 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:58.478073 6116 tablet_bootstrap.cc:492] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Bootstrap complete.
I20250112 20:51:58.479640 6116 ts_tablet_manager.cc:1397] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Time spent bootstrapping tablet: real 0.047s user 0.036s sys 0.008s
I20250112 20:51:58.481060 6094 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46001
I20250112 20:51:58.481159 6230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46001 every 8 connection(s)
I20250112 20:51:58.482314 6094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnWriteOpsITest.TxnWriteAfterCommit.1736715060603202-416-0/minicluster-data/ts-2/data/info.pb
I20250112 20:51:58.484231 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskRSg17H/build/asan/bin/kudu as pid 6094
I20250112 20:51:58.484875 6231 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35703
I20250112 20:51:58.485141 6231 heartbeater.cc:463] Registering TS with master...
I20250112 20:51:58.486119 6231 heartbeater.cc:510] Master 127.0.104.62:35703 requested a full tablet report, sending...
I20250112 20:51:58.486276 6116 raft_consensus.cc:357] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:58.487205 6116 raft_consensus.cc:738] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737fa552528642308bfd0388e079ad88, State: Initialized, Role: FOLLOWER
I20250112 20:51:58.487831 6116 consensus_queue.cc:260] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:58.491454 6116 ts_tablet_manager.cc:1428] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20250112 20:51:58.491493 5253 ts_manager.cc:194] Re-registered known tserver with Master: 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:58.491925 6116 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Bootstrap starting.
I20250112 20:51:58.494102 5253 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:36641
I20250112 20:51:58.497025 6231 heartbeater.cc:502] Master 127.0.104.62:35703 was elected leader, sending a full tablet report...
I20250112 20:51:58.505471 6116 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:58.506003 6116 tablet_bootstrap.cc:492] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Bootstrap complete.
I20250112 20:51:58.506911 6116 ts_tablet_manager.cc:1397] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250112 20:51:58.507539 6116 raft_consensus.cc:357] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:58.507715 6116 raft_consensus.cc:738] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737fa552528642308bfd0388e079ad88, State: Initialized, Role: FOLLOWER
I20250112 20:51:58.507947 6116 consensus_queue.cc:260] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:58.508381 6116 ts_tablet_manager.cc:1428] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250112 20:51:58.508646 6116 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Bootstrap starting.
I20250112 20:51:58.529078 6116 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=4 ignored=0} mutations{seen=6 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:51:58.529558 6116 tablet_bootstrap.cc:492] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Bootstrap complete.
I20250112 20:51:58.530517 6116 ts_tablet_manager.cc:1397] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.004s
I20250112 20:51:58.531260 6116 raft_consensus.cc:357] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:58.531425 6116 raft_consensus.cc:738] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737fa552528642308bfd0388e079ad88, State: Initialized, Role: FOLLOWER
I20250112 20:51:58.531674 6116 consensus_queue.cc:260] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:58.532196 6116 ts_tablet_manager.cc:1428] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:51:58.532128 6235 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:58.532503 6235 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:51:59.112402 6242 raft_consensus.cc:491] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 737fa552528642308bfd0388e079ad88)
I20250112 20:51:59.112701 6242 raft_consensus.cc:513] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:59.114215 6242 leader_election.cc:290] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:59.127875 6247 raft_consensus.cc:491] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:51:59.128238 6247 raft_consensus.cc:513] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:59.129985 6247 leader_election.cc:290] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889), 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:59.132814 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "737fa552528642308bfd0388e079ad88" is_pre_election: true
I20250112 20:51:59.132824 6040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d" is_pre_election: true
I20250112 20:51:59.133323 6186 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 1.
I20250112 20:51:59.133359 6040 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 1.
I20250112 20:51:59.133951 5820 leader_election.cc:304] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737fa552528642308bfd0388e079ad88, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:59.134518 6242 raft_consensus.cc:2798] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 20:51:59.134707 6242 raft_consensus.cc:491] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 737fa552528642308bfd0388e079ad88)
I20250112 20:51:59.134804 6242 raft_consensus.cc:3054] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.138988 6242 raft_consensus.cc:513] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:59.139911 6242 leader_election.cc:290] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 election: Requested vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:59.140199 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "737fa552528642308bfd0388e079ad88"
I20250112 20:51:59.140184 6040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d"
I20250112 20:51:59.140394 6040 raft_consensus.cc:3054] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.140394 6186 raft_consensus.cc:3054] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.145936 6186 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 2.
I20250112 20:51:59.146040 6040 raft_consensus.cc:2463] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 2.
I20250112 20:51:59.146389 5820 leader_election.cc:304] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737fa552528642308bfd0388e079ad88, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:59.146754 6242 raft_consensus.cc:2798] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:51:59.146989 5896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "1cdfc7648c274dec813577a70a67e12d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ea410e5f892a427b893e5f4729675c1a" is_pre_election: true
I20250112 20:51:59.147383 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de9ddce96cce47f7aaeb0dfa4db65ad3" candidate_uuid: "1cdfc7648c274dec813577a70a67e12d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "737fa552528642308bfd0388e079ad88" is_pre_election: true
I20250112 20:51:59.147785 6186 raft_consensus.cc:2388] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 737fa552528642308bfd0388e079ad88 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cdfc7648c274dec813577a70a67e12d in current term 2: Already voted for candidate ea410e5f892a427b893e5f4729675c1a in this term.
I20250112 20:51:59.148450 6242 raft_consensus.cc:695] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [term 2 LEADER]: Becoming Leader. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Running, Role: LEADER
I20250112 20:51:59.148995 6242 consensus_queue.cc:237] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:59.152387 5966 leader_election.cc:304] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d; no voters: 737fa552528642308bfd0388e079ad88, ea410e5f892a427b893e5f4729675c1a
I20250112 20:51:59.152792 6247 raft_consensus.cc:2743] T de9ddce96cce47f7aaeb0dfa4db65ad3 P 1cdfc7648c274dec813577a70a67e12d [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250112 20:51:59.153499 5252 catalog_manager.cc:5526] T de9ddce96cce47f7aaeb0dfa4db65ad3 P ea410e5f892a427b893e5f4729675c1a reported cstate change: term changed from 1 to 2, leader changed from 737fa552528642308bfd0388e079ad88 (127.0.104.3) to ea410e5f892a427b893e5f4729675c1a (127.0.104.1). New cstate: current_term: 2 leader_uuid: "ea410e5f892a427b893e5f4729675c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } health_report { overall_health: UNKNOWN } } }
I20250112 20:51:59.323355 6242 raft_consensus.cc:491] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 737fa552528642308bfd0388e079ad88)
I20250112 20:51:59.323586 6242 raft_consensus.cc:513] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:59.324270 6242 leader_election.cc:290] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:59.324940 6040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d" is_pre_election: true
I20250112 20:51:59.324965 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "737fa552528642308bfd0388e079ad88" is_pre_election: true
I20250112 20:51:59.325181 6040 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 1.
I20250112 20:51:59.325253 6186 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 1.
I20250112 20:51:59.325634 5821 leader_election.cc:304] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:59.325958 6242 raft_consensus.cc:2798] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 20:51:59.326053 6242 raft_consensus.cc:491] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 737fa552528642308bfd0388e079ad88)
I20250112 20:51:59.326176 6242 raft_consensus.cc:3054] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.328231 6242 raft_consensus.cc:513] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:59.328938 6242 leader_election.cc:290] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 election: Requested vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:59.329334 6040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d"
I20250112 20:51:59.329488 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8f9ac7de6d4211bf4ff89b7ca5acbf" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "737fa552528642308bfd0388e079ad88"
I20250112 20:51:59.329572 6040 raft_consensus.cc:3054] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.329737 6186 raft_consensus.cc:3054] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.332373 6040 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 1cdfc7648c274dec813577a70a67e12d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 2.
I20250112 20:51:59.332373 6186 raft_consensus.cc:2463] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P 737fa552528642308bfd0388e079ad88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 2.
I20250112 20:51:59.332832 5821 leader_election.cc:304] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:59.333122 6242 raft_consensus.cc:2798] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:51:59.333310 6242 raft_consensus.cc:695] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [term 2 LEADER]: Becoming Leader. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Running, Role: LEADER
I20250112 20:51:59.333753 6242 consensus_queue.cc:237] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } }
I20250112 20:51:59.336361 5252 catalog_manager.cc:5526] T 9b8f9ac7de6d4211bf4ff89b7ca5acbf P ea410e5f892a427b893e5f4729675c1a reported cstate change: term changed from 1 to 2, leader changed from 737fa552528642308bfd0388e079ad88 (127.0.104.3) to ea410e5f892a427b893e5f4729675c1a (127.0.104.1). New cstate: current_term: 2 leader_uuid: "ea410e5f892a427b893e5f4729675c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } health_report { overall_health: HEALTHY } } }
I20250112 20:51:59.469883 6242 raft_consensus.cc:491] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 737fa552528642308bfd0388e079ad88)
I20250112 20:51:59.470075 6242 raft_consensus.cc:513] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:59.470793 6242 leader_election.cc:290] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:59.471213 6040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d" is_pre_election: true
I20250112 20:51:59.471310 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "737fa552528642308bfd0388e079ad88" is_pre_election: true
I20250112 20:51:59.471433 6040 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 1.
I20250112 20:51:59.471553 6186 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 1.
I20250112 20:51:59.471894 5821 leader_election.cc:304] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:59.472167 6242 raft_consensus.cc:2798] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250112 20:51:59.472303 6242 raft_consensus.cc:491] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 737fa552528642308bfd0388e079ad88)
I20250112 20:51:59.472391 6242 raft_consensus.cc:3054] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.474340 6242 raft_consensus.cc:513] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:59.474917 6242 leader_election.cc:290] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 election: Requested vote from peers 1cdfc7648c274dec813577a70a67e12d (127.0.104.2:38157), 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001)
I20250112 20:51:59.475299 6040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "1cdfc7648c274dec813577a70a67e12d"
I20250112 20:51:59.475441 6186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bbd5f1d33c746a687c1892773195f1d" candidate_uuid: "ea410e5f892a427b893e5f4729675c1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "737fa552528642308bfd0388e079ad88"
I20250112 20:51:59.475512 6040 raft_consensus.cc:3054] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.475627 6186 raft_consensus.cc:3054] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:51:59.477404 6040 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P 1cdfc7648c274dec813577a70a67e12d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 2.
I20250112 20:51:59.477497 6186 raft_consensus.cc:2463] T 1bbd5f1d33c746a687c1892773195f1d P 737fa552528642308bfd0388e079ad88 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea410e5f892a427b893e5f4729675c1a in term 2.
I20250112 20:51:59.477803 5821 leader_election.cc:304] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdfc7648c274dec813577a70a67e12d, ea410e5f892a427b893e5f4729675c1a; no voters:
I20250112 20:51:59.478113 6242 raft_consensus.cc:2798] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:51:59.478260 6242 raft_consensus.cc:695] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [term 2 LEADER]: Becoming Leader. State: Replica: ea410e5f892a427b893e5f4729675c1a, State: Running, Role: LEADER
I20250112 20:51:59.478550 6242 consensus_queue.cc:237] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } }
I20250112 20:51:59.479940 6252 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: New leader ea410e5f892a427b893e5f4729675c1a. Latest consensus state: current_term: 2 leader_uuid: "ea410e5f892a427b893e5f4729675c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } } }
I20250112 20:51:59.480170 6252 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250112 20:51:59.485046 6260 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:51:59.485352 6260 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:51:59.485563 5252 catalog_manager.cc:5526] T 1bbd5f1d33c746a687c1892773195f1d P ea410e5f892a427b893e5f4729675c1a reported cstate change: term changed from 1 to 2, leader changed from 737fa552528642308bfd0388e079ad88 (127.0.104.3) to ea410e5f892a427b893e5f4729675c1a (127.0.104.1). New cstate: current_term: 2 leader_uuid: "ea410e5f892a427b893e5f4729675c1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea410e5f892a427b893e5f4729675c1a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cdfc7648c274dec813577a70a67e12d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "737fa552528642308bfd0388e079ad88" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46001 } health_report { overall_health: UNKNOWN } } }
W20250112 20:51:59.515152 6236 tablet_replica.cc:1307] Illegal state: Failed to write to server: ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889): transaction ID 1 not open: COMMITTED: cancelling pending write operations
W20250112 20:51:59.515431 6236 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44026: Illegal state: Failed to write to server: ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889): transaction ID 1 not open: COMMITTED
I20250112 20:51:59.515837 6236 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:44026 (ReqId={client: 413fe5c7c384471c96beb7c473215c4e, seq_no=3, attempt_no=0}) took 1021 ms. Trace:
I20250112 20:51:59.515956 6236 rpcz_store.cc:276] 0112 20:51:58.494784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:51:58.494900 (+ 116us) service_pool.cc:225] Handling call
0112 20:51:59.515792 (+1020892us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":348,"threads_started":1,"txn-participant-registration.queue_time_us":513}
W20250112 20:51:59.517204 5732 batcher.cc:441] Illegal state: Failed to write batch of 1 ops to tablet 9b8f9ac7de6d4211bf4ff89b7ca5acbf after 1 attempt(s): Failed to write to server: 737fa552528642308bfd0388e079ad88 (127.0.104.3:46001): Failed to write to server: ea410e5f892a427b893e5f4729675c1a (127.0.104.1:34889): transaction ID 1 not open: COMMITTED
I20250112 20:51:59.518289 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 5801
I20250112 20:51:59.526108 5936 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:59.624106 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 5801
I20250112 20:51:59.645040 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 5948
I20250112 20:51:59.651481 6081 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:59.740177 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 5948
I20250112 20:51:59.758481 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 6094
I20250112 20:51:59.765121 6226 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:59.856465 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 6094
I20250112 20:51:59.873991 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskRSg17H/build/asan/bin/kudu pid 5222
I20250112 20:51:59.875095 5293 generic_service.cc:196] Checking for leaks (request via RPC)
I20250112 20:51:59.984922 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskRSg17H/build/asan/bin/kudu with pid 5222
2025-01-12T20:52:00Z chronyd exiting
[ OK ] TxnWriteOpsITest.TxnWriteAfterCommit (6086 ms)
[----------] 8 tests from TxnWriteOpsITest (59348 ms total)
[----------] 18 tests from TxnOpDispatcherITest
[ RUN ] TxnOpDispatcherITest.LifecycleBasic
I20250112 20:52:00.029094 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:35545
I20250112 20:52:00.030369 416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250112 20:52:00.031338 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:00.037760 6269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:00.037950 6270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:00.038352 6272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:00.039796 416 server_base.cc:1034] running on GCE node
I20250112 20:52:00.040608 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:00.040804 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:00.040907 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715120040867 us; error 0 us; skew 500 ppm
I20250112 20:52:00.041285 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:00.043195 416 webserver.cc:458] Webserver started at http://127.0.104.62:33715/ using document root <none> and password file <none>
I20250112 20:52:00.044008 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:00.044143 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:00.044443 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:00.046752 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "109fa336f6ab49d9a17c02f883ade6bb"
format_stamp: "Formatted at 2025-01-12 20:52:00 on dist-test-slave-3gql"
I20250112 20:52:00.052006 416 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250112 20:52:00.055400 6277 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:00.056468 416 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250112 20:52:00.056666 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "109fa336f6ab49d9a17c02f883ade6bb"
format_stamp: "Formatted at 2025-01-12 20:52:00 on dist-test-slave-3gql"
I20250112 20:52:00.056978 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:00.077826 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:00.078872 416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250112 20:52:00.079159 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:00.102017 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35545
I20250112 20:52:00.102038 6338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35545 every 8 connection(s)
I20250112 20:52:00.104946 6339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:00.115168 6339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb: Bootstrap starting.
I20250112 20:52:00.118472 6339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:00.119644 6339 log.cc:826] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb: Log is configured to *not* fsync() on all Append() calls
I20250112 20:52:00.122242 6339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb: No bootstrap required, opened a new log
I20250112 20:52:00.126955 6339 raft_consensus.cc:357] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109fa336f6ab49d9a17c02f883ade6bb" member_type: VOTER }
I20250112 20:52:00.127276 6339 raft_consensus.cc:383] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:00.127388 6339 raft_consensus.cc:738] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 109fa336f6ab49d9a17c02f883ade6bb, State: Initialized, Role: FOLLOWER
I20250112 20:52:00.128202 6339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109fa336f6ab49d9a17c02f883ade6bb" member_type: VOTER }
I20250112 20:52:00.128451 6339 raft_consensus.cc:397] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:00.128621 6339 raft_consensus.cc:491] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:00.128819 6339 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:00.131397 6339 raft_consensus.cc:513] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109fa336f6ab49d9a17c02f883ade6bb" member_type: VOTER }
I20250112 20:52:00.131938 6339 leader_election.cc:304] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 109fa336f6ab49d9a17c02f883ade6bb; no voters:
I20250112 20:52:00.132560 6339 leader_election.cc:290] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:00.132746 6342 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:00.133311 6342 raft_consensus.cc:695] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 1 LEADER]: Becoming Leader. State: Replica: 109fa336f6ab49d9a17c02f883ade6bb, State: Running, Role: LEADER
I20250112 20:52:00.133862 6342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109fa336f6ab49d9a17c02f883ade6bb" member_type: VOTER }
I20250112 20:52:00.134594 6339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:00.136915 6343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "109fa336f6ab49d9a17c02f883ade6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109fa336f6ab49d9a17c02f883ade6bb" member_type: VOTER } }
I20250112 20:52:00.137224 6343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:00.137553 6344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 109fa336f6ab49d9a17c02f883ade6bb. Latest consensus state: current_term: 1 leader_uuid: "109fa336f6ab49d9a17c02f883ade6bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "109fa336f6ab49d9a17c02f883ade6bb" member_type: VOTER } }
I20250112 20:52:00.137777 6344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:00.139695 6349 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:00.145471 6349 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:00.146044 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:00.155618 6349 catalog_manager.cc:1348] Generated new cluster ID: 935e871bffe846618e7c2683150ae61a
I20250112 20:52:00.155750 6349 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:00.173794 6349 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:00.175189 6349 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:00.186674 6349 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb: Generated new TSK 0
I20250112 20:52:00.187494 6349 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:00.197849 6294 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34964:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:52:00.205487 6339 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 1; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
I20250112 20:52:00.211134 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:00.214906 6367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:00.215341 6368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:00.216948 416 server_base.cc:1034] running on GCE node
W20250112 20:52:00.217120 6370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:00.217504 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:00.217576 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:00.217661 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715120217614 us; error 0 us; skew 500 ppm
I20250112 20:52:00.217834 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:00.219314 416 webserver.cc:458] Webserver started at http://127.0.104.1:35769/ using document root <none> and password file <none>
I20250112 20:52:00.219561 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:00.219671 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:00.219836 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:00.220584 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "13a9a4961bf34a848ea53a96a43b20ae"
format_stamp: "Formatted at 2025-01-12 20:52:00 on dist-test-slave-3gql"
I20250112 20:52:00.223373 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250112 20:52:00.225479 6375 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:00.225912 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:00.226051 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "13a9a4961bf34a848ea53a96a43b20ae"
format_stamp: "Formatted at 2025-01-12 20:52:00 on dist-test-slave-3gql"
I20250112 20:52:00.226238 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.LifecycleBasic.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:00.253271 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:00.254124 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:00.257812 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:00.258033 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:00.258260 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:00.258363 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:00.307044 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46613
I20250112 20:52:00.307117 6444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46613 every 8 connection(s)
I20250112 20:52:00.309597 6445 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35545
I20250112 20:52:00.309929 6445 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:00.310778 6445 heartbeater.cc:510] Master 127.0.104.62:35545 requested a full tablet report, sending...
I20250112 20:52:00.312834 6294 ts_manager.cc:194] Registered new tserver with Master: 13a9a4961bf34a848ea53a96a43b20ae (127.0.104.1:46613)
I20250112 20:52:00.313628 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.004606019s
I20250112 20:52:00.315006 6294 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34968
I20250112 20:52:00.327580 6294 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34986:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:00.349339 6409 tablet_service.cc:1467] Processing CreateTablet for tablet f78a4d0082a243eeac9dc2a5efa11d7c (DEFAULT_TABLE table=txn_op_dispatcher_test [id=30cdeae5c55c4e8eabdc67274320398d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:00.349366 6410 tablet_service.cc:1467] Processing CreateTablet for tablet d3324c97c6cf40bda5d6795868b844f2 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=30cdeae5c55c4e8eabdc67274320398d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:00.350111 6409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f78a4d0082a243eeac9dc2a5efa11d7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:00.350404 6410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3324c97c6cf40bda5d6795868b844f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:00.357264 6455 tablet_bootstrap.cc:492] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae: Bootstrap starting.
I20250112 20:52:00.359189 6455 tablet_bootstrap.cc:654] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:00.361151 6455 tablet_bootstrap.cc:492] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae: No bootstrap required, opened a new log
I20250112 20:52:00.361318 6455 ts_tablet_manager.cc:1397] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250112 20:52:00.362249 6455 raft_consensus.cc:357] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.362468 6455 raft_consensus.cc:383] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:00.362603 6455 raft_consensus.cc:738] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a9a4961bf34a848ea53a96a43b20ae, State: Initialized, Role: FOLLOWER
I20250112 20:52:00.362847 6455 consensus_queue.cc:260] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.363027 6455 raft_consensus.cc:397] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:00.363134 6455 raft_consensus.cc:491] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:00.363241 6455 raft_consensus.cc:3054] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:00.365280 6455 raft_consensus.cc:513] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.365553 6455 leader_election.cc:304] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a9a4961bf34a848ea53a96a43b20ae; no voters:
I20250112 20:52:00.366081 6455 leader_election.cc:290] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:00.366276 6457 raft_consensus.cc:2798] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:00.366954 6455 ts_tablet_manager.cc:1428] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20250112 20:52:00.367173 6445 heartbeater.cc:502] Master 127.0.104.62:35545 was elected leader, sending a full tablet report...
I20250112 20:52:00.367305 6455 tablet_bootstrap.cc:492] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae: Bootstrap starting.
I20250112 20:52:00.368791 6455 tablet_bootstrap.cc:654] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:00.370133 6457 raft_consensus.cc:695] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 1 LEADER]: Becoming Leader. State: Replica: 13a9a4961bf34a848ea53a96a43b20ae, State: Running, Role: LEADER
I20250112 20:52:00.370579 6457 consensus_queue.cc:237] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.374433 6455 tablet_bootstrap.cc:492] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae: No bootstrap required, opened a new log
I20250112 20:52:00.374629 6455 ts_tablet_manager.cc:1397] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20250112 20:52:00.375295 6294 catalog_manager.cc:5526] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a9a4961bf34a848ea53a96a43b20ae (127.0.104.1). New cstate: current_term: 1 leader_uuid: "13a9a4961bf34a848ea53a96a43b20ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:00.375703 6455 raft_consensus.cc:357] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.375933 6455 raft_consensus.cc:383] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:00.376039 6455 raft_consensus.cc:738] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a9a4961bf34a848ea53a96a43b20ae, State: Initialized, Role: FOLLOWER
I20250112 20:52:00.376427 6455 consensus_queue.cc:260] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.376708 6455 raft_consensus.cc:397] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:00.376930 6455 raft_consensus.cc:491] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:00.377056 6455 raft_consensus.cc:3054] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:00.380441 6455 raft_consensus.cc:513] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.380790 6455 leader_election.cc:304] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a9a4961bf34a848ea53a96a43b20ae; no voters:
I20250112 20:52:00.381145 6457 raft_consensus.cc:2798] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:00.384090 6455 leader_election.cc:290] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:00.384286 6457 raft_consensus.cc:695] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 1 LEADER]: Becoming Leader. State: Replica: 13a9a4961bf34a848ea53a96a43b20ae, State: Running, Role: LEADER
I20250112 20:52:00.384536 6457 consensus_queue.cc:237] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:00.385183 6455 ts_tablet_manager.cc:1428] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20250112 20:52:00.387557 6294 catalog_manager.cc:5526] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a9a4961bf34a848ea53a96a43b20ae (127.0.104.1). New cstate: current_term: 1 leader_uuid: "13a9a4961bf34a848ea53a96a43b20ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:00.406208 6450 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:01.227450 6294 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34998:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:01.234881 6409 tablet_service.cc:1467] Processing CreateTablet for tablet 59c6f03d65844f2d81fe7f66b1e5b3df (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4a4834f8046f4f189a49d7ed9dc24af8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:01.235384 6409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c6f03d65844f2d81fe7f66b1e5b3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:01.240046 6473 tablet_bootstrap.cc:492] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: Bootstrap starting.
I20250112 20:52:01.241668 6473 tablet_bootstrap.cc:654] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:01.243260 6473 tablet_bootstrap.cc:492] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: No bootstrap required, opened a new log
I20250112 20:52:01.243393 6473 ts_tablet_manager.cc:1397] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:52:01.244232 6473 raft_consensus.cc:357] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:01.244378 6473 raft_consensus.cc:383] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:01.244444 6473 raft_consensus.cc:738] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13a9a4961bf34a848ea53a96a43b20ae, State: Initialized, Role: FOLLOWER
I20250112 20:52:01.244686 6473 consensus_queue.cc:260] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:01.244830 6473 raft_consensus.cc:397] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:01.244887 6473 raft_consensus.cc:491] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:01.244964 6473 raft_consensus.cc:3054] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:01.246544 6473 raft_consensus.cc:513] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:01.246743 6473 leader_election.cc:304] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13a9a4961bf34a848ea53a96a43b20ae; no voters:
I20250112 20:52:01.247078 6473 leader_election.cc:290] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:01.247232 6475 raft_consensus.cc:2798] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:01.248003 6473 ts_tablet_manager.cc:1428] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250112 20:52:01.248384 6475 raft_consensus.cc:695] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 1 LEADER]: Becoming Leader. State: Replica: 13a9a4961bf34a848ea53a96a43b20ae, State: Running, Role: LEADER
I20250112 20:52:01.248679 6475 consensus_queue.cc:237] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } }
I20250112 20:52:01.250057 6476 tablet_replica.cc:440] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13a9a4961bf34a848ea53a96a43b20ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } } }
I20250112 20:52:01.250352 6476 tablet_replica.cc:443] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:01.250283 6477 tablet_replica.cc:440] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: TxnStatusTablet state changed. Reason: New leader 13a9a4961bf34a848ea53a96a43b20ae. Latest consensus state: current_term: 1 leader_uuid: "13a9a4961bf34a848ea53a96a43b20ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } } }
I20250112 20:52:01.250553 6477 tablet_replica.cc:443] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:01.251310 6479 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:01.251559 6479 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:01.251686 6294 catalog_manager.cc:5526] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 13a9a4961bf34a848ea53a96a43b20ae (127.0.104.1). New cstate: current_term: 1 leader_uuid: "13a9a4961bf34a848ea53a96a43b20ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13a9a4961bf34a848ea53a96a43b20ae" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46613 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:01.484242 6493 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250112 20:52:01.494347 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:01.508834 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:01.509518 416 tablet_replica.cc:331] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae: stopping tablet replica
I20250112 20:52:01.509999 416 raft_consensus.cc:2238] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:01.510316 416 raft_consensus.cc:2267] T 59c6f03d65844f2d81fe7f66b1e5b3df P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:01.512099 416 tablet_replica.cc:331] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae: stopping tablet replica
I20250112 20:52:01.512276 416 raft_consensus.cc:2238] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:01.512483 416 raft_consensus.cc:2267] T f78a4d0082a243eeac9dc2a5efa11d7c P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:01.513506 416 tablet_replica.cc:331] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae: stopping tablet replica
I20250112 20:52:01.513692 416 raft_consensus.cc:2238] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:01.513896 416 raft_consensus.cc:2267] T d3324c97c6cf40bda5d6795868b844f2 P 13a9a4961bf34a848ea53a96a43b20ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:01.531234 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:01.538239 416 master.cc:537] Master@127.0.104.62:35545 shutting down...
I20250112 20:52:01.551695 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:01.551973 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:01.552134 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 109fa336f6ab49d9a17c02f883ade6bb: stopping tablet replica
I20250112 20:52:01.557273 416 master.cc:559] Master@127.0.104.62:35545 shutdown complete.
[ OK ] TxnOpDispatcherITest.LifecycleBasic (1540 ms)
[ RUN ] TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable
I20250112 20:52:01.569705 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:46581
I20250112 20:52:01.570294 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:01.572739 6496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:01.572896 6495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:01.574116 416 server_base.cc:1034] running on GCE node
W20250112 20:52:01.574287 6499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:01.574693 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:01.574764 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:01.574813 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715121574788 us; error 0 us; skew 500 ppm
I20250112 20:52:01.575031 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:01.576610 416 webserver.cc:458] Webserver started at http://127.0.104.62:37773/ using document root <none> and password file <none>
I20250112 20:52:01.576902 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:01.576985 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:01.577277 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:01.578087 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "62a56af2da7b4bff9e4dbdd101d5d080"
format_stamp: "Formatted at 2025-01-12 20:52:01 on dist-test-slave-3gql"
I20250112 20:52:01.581133 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250112 20:52:01.583184 6503 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:01.583638 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250112 20:52:01.583778 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "62a56af2da7b4bff9e4dbdd101d5d080"
format_stamp: "Formatted at 2025-01-12 20:52:01 on dist-test-slave-3gql"
I20250112 20:52:01.583946 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:01.601073 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:01.601962 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:01.647140 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46581
I20250112 20:52:01.647210 6564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46581 every 8 connection(s)
I20250112 20:52:01.649303 6565 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:01.656980 6565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080: Bootstrap starting.
I20250112 20:52:01.658762 6565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:01.660710 6565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080: No bootstrap required, opened a new log
I20250112 20:52:01.661496 6565 raft_consensus.cc:357] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" member_type: VOTER }
I20250112 20:52:01.661665 6565 raft_consensus.cc:383] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:01.661779 6565 raft_consensus.cc:738] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62a56af2da7b4bff9e4dbdd101d5d080, State: Initialized, Role: FOLLOWER
I20250112 20:52:01.662055 6565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" member_type: VOTER }
I20250112 20:52:01.662232 6565 raft_consensus.cc:397] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:01.662336 6565 raft_consensus.cc:491] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:01.662460 6565 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:01.664211 6565 raft_consensus.cc:513] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" member_type: VOTER }
I20250112 20:52:01.664444 6565 leader_election.cc:304] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62a56af2da7b4bff9e4dbdd101d5d080; no voters:
I20250112 20:52:01.664949 6565 leader_election.cc:290] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:01.665079 6568 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:01.665625 6568 raft_consensus.cc:695] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 1 LEADER]: Becoming Leader. State: Replica: 62a56af2da7b4bff9e4dbdd101d5d080, State: Running, Role: LEADER
I20250112 20:52:01.666023 6568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" member_type: VOTER }
I20250112 20:52:01.667011 6568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62a56af2da7b4bff9e4dbdd101d5d080. Latest consensus state: current_term: 1 leader_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" member_type: VOTER } }
I20250112 20:52:01.667137 6570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62a56af2da7b4bff9e4dbdd101d5d080" member_type: VOTER } }
I20250112 20:52:01.667261 6568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:01.667340 6570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:01.672360 6565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:01.675089 6572 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:01.677150 6572 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:01.681967 6572 catalog_manager.cc:1348] Generated new cluster ID: 8c1a4bf8f72d4caeba4b6ac8d1a2adde
I20250112 20:52:01.682090 6572 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:01.686221 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:01.691846 6572 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:01.692919 6572 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:01.729197 6572 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080: Generated new TSK 0
I20250112 20:52:01.729514 6572 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:01.741657 6520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41724:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:01.751175 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:01.754942 6592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:01.754961 6593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:01.755472 6595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:01.755616 416 server_base.cc:1034] running on GCE node
I20250112 20:52:01.755985 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:01.756050 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:01.756108 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715121756076 us; error 0 us; skew 500 ppm
I20250112 20:52:01.756268 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:01.757548 416 webserver.cc:458] Webserver started at http://127.0.104.1:38667/ using document root <none> and password file <none>
I20250112 20:52:01.757776 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:01.757869 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:01.758014 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:01.758635 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "7c904621a4004a318d1780673619c909"
format_stamp: "Formatted at 2025-01-12 20:52:01 on dist-test-slave-3gql"
I20250112 20:52:01.761054 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250112 20:52:01.762777 6600 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:01.763193 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:01.763288 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "7c904621a4004a318d1780673619c909"
format_stamp: "Formatted at 2025-01-12 20:52:01 on dist-test-slave-3gql"
I20250112 20:52:01.763406 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:01.774693 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:01.775292 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:01.777029 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:01.777160 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:01.777364 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:01.777467 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250112 20:52:01.806380 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40715
I20250112 20:52:01.806414 6669 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40715 every 8 connection(s)
I20250112 20:52:01.808295 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 20:52:01.808319 6670 heartbeater.cc:346] Connected to a master server at 127.0.104.62:46581
I20250112 20:52:01.808444 6670 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:01.808931 6670 heartbeater.cc:510] Master 127.0.104.62:46581 requested a full tablet report, sending...
I20250112 20:52:01.809868 6520 ts_manager.cc:194] Registered new tserver with Master: 7c904621a4004a318d1780673619c909 (127.0.104.1:40715)
W20250112 20:52:01.811359 6672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:01.811679 6673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:01.812830 6520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41730
W20250112 20:52:01.814134 6675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:01.814137 416 server_base.cc:1034] running on GCE node
I20250112 20:52:01.814504 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:01.814569 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:01.814625 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715121814600 us; error 0 us; skew 500 ppm
I20250112 20:52:01.814764 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:01.816033 416 webserver.cc:458] Webserver started at http://127.0.104.2:38653/ using document root <none> and password file <none>
I20250112 20:52:01.816241 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:01.816322 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:01.816432 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:01.817080 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-1-root/instance:
uuid: "794ff08a1c4249649eb1c93b804b6aae"
format_stamp: "Formatted at 2025-01-12 20:52:01 on dist-test-slave-3gql"
I20250112 20:52:01.819443 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250112 20:52:01.820923 6680 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:01.821257 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:01.821358 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-1-root
uuid: "794ff08a1c4249649eb1c93b804b6aae"
format_stamp: "Formatted at 2025-01-12 20:52:01 on dist-test-slave-3gql"
I20250112 20:52:01.821489 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:01.827994 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:01.828580 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:01.830268 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:01.830397 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:01.830525 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:01.830636 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:01.856887 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:34777
I20250112 20:52:01.856956 6749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:34777 every 8 connection(s)
I20250112 20:52:01.858587 6750 heartbeater.cc:346] Connected to a master server at 127.0.104.62:46581
I20250112 20:52:01.858712 6750 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:01.859107 6750 heartbeater.cc:510] Master 127.0.104.62:46581 requested a full tablet report, sending...
I20250112 20:52:01.859836 6520 ts_manager.cc:194] Registered new tserver with Master: 794ff08a1c4249649eb1c93b804b6aae (127.0.104.2:34777)
I20250112 20:52:01.860205 416 internal_mini_cluster.cc:371] 2 TS(s) registered with all masters after 0.002353552s
I20250112 20:52:01.860677 6520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41754
I20250112 20:52:02.762140 6520 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41788:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:02.775193 6635 tablet_service.cc:1467] Processing CreateTablet for tablet f425f21d85344434b50921b41b60dc84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fba820afec2d4ac2b1ac2ac0f8fa9d33]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:02.775730 6635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f425f21d85344434b50921b41b60dc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:02.780485 6770 tablet_bootstrap.cc:492] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Bootstrap starting.
I20250112 20:52:02.782096 6770 tablet_bootstrap.cc:654] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:02.783967 6770 tablet_bootstrap.cc:492] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: No bootstrap required, opened a new log
I20250112 20:52:02.784116 6770 ts_tablet_manager.cc:1397] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250112 20:52:02.784843 6770 raft_consensus.cc:357] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:02.785056 6770 raft_consensus.cc:383] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:02.785183 6770 raft_consensus.cc:738] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c904621a4004a318d1780673619c909, State: Initialized, Role: FOLLOWER
I20250112 20:52:02.785415 6770 consensus_queue.cc:260] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:02.785590 6770 raft_consensus.cc:397] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:02.785662 6770 raft_consensus.cc:491] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:02.785794 6770 raft_consensus.cc:3054] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:02.787509 6770 raft_consensus.cc:513] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:02.787739 6770 leader_election.cc:304] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c904621a4004a318d1780673619c909; no voters:
I20250112 20:52:02.788216 6770 leader_election.cc:290] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:02.788364 6772 raft_consensus.cc:2798] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:02.788981 6670 heartbeater.cc:502] Master 127.0.104.62:46581 was elected leader, sending a full tablet report...
I20250112 20:52:02.788946 6772 raft_consensus.cc:695] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 LEADER]: Becoming Leader. State: Replica: 7c904621a4004a318d1780673619c909, State: Running, Role: LEADER
I20250112 20:52:02.789268 6772 consensus_queue.cc:237] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:02.789516 6770 ts_tablet_manager.cc:1428] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
I20250112 20:52:02.790078 6773 tablet_replica.cc:440] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c904621a4004a318d1780673619c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } } }
I20250112 20:52:02.790148 6774 tablet_replica.cc:440] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: TxnStatusTablet state changed. Reason: New leader 7c904621a4004a318d1780673619c909. Latest consensus state: current_term: 1 leader_uuid: "7c904621a4004a318d1780673619c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } } }
I20250112 20:52:02.790467 6774 tablet_replica.cc:443] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:02.790470 6773 tablet_replica.cc:443] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:02.791352 6776 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:02.791491 6776 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:02.791615 6520 catalog_manager.cc:5526] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c904621a4004a318d1780673619c909 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "7c904621a4004a318d1780673619c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:02.862124 6750 heartbeater.cc:502] Master 127.0.104.62:46581 was elected leader, sending a full tablet report...
I20250112 20:52:02.897521 6519 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41772:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250112 20:52:02.909853 6714 tablet_service.cc:1467] Processing CreateTablet for tablet adc3c3b0046c4528b7b3fd2962478478 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=03510b5b25d344f59059f6b0031f414e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:02.910383 6714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adc3c3b0046c4528b7b3fd2962478478. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:02.915217 6779 tablet_bootstrap.cc:492] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae: Bootstrap starting.
I20250112 20:52:02.916728 6779 tablet_bootstrap.cc:654] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:02.918581 6779 tablet_bootstrap.cc:492] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae: No bootstrap required, opened a new log
I20250112 20:52:02.918699 6779 ts_tablet_manager.cc:1397] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250112 20:52:02.919296 6779 raft_consensus.cc:357] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794ff08a1c4249649eb1c93b804b6aae" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34777 } }
I20250112 20:52:02.919433 6779 raft_consensus.cc:383] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:02.919497 6779 raft_consensus.cc:738] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 794ff08a1c4249649eb1c93b804b6aae, State: Initialized, Role: FOLLOWER
I20250112 20:52:02.919668 6779 consensus_queue.cc:260] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794ff08a1c4249649eb1c93b804b6aae" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34777 } }
I20250112 20:52:02.919814 6779 raft_consensus.cc:397] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:02.919867 6779 raft_consensus.cc:491] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:02.919946 6779 raft_consensus.cc:3054] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:02.921618 6779 raft_consensus.cc:513] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794ff08a1c4249649eb1c93b804b6aae" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34777 } }
I20250112 20:52:02.921811 6779 leader_election.cc:304] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 794ff08a1c4249649eb1c93b804b6aae; no voters:
I20250112 20:52:02.922228 6779 leader_election.cc:290] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:02.922454 6781 raft_consensus.cc:2798] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:02.923264 6779 ts_tablet_manager.cc:1428] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20250112 20:52:02.923609 6781 raft_consensus.cc:695] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 1 LEADER]: Becoming Leader. State: Replica: 794ff08a1c4249649eb1c93b804b6aae, State: Running, Role: LEADER
I20250112 20:52:02.923872 6781 consensus_queue.cc:237] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794ff08a1c4249649eb1c93b804b6aae" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34777 } }
I20250112 20:52:02.926024 6519 catalog_manager.cc:5526] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae reported cstate change: term changed from 0 to 1, leader changed from <none> to 794ff08a1c4249649eb1c93b804b6aae (127.0.104.2). New cstate: current_term: 1 leader_uuid: "794ff08a1c4249649eb1c93b804b6aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "794ff08a1c4249649eb1c93b804b6aae" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34777 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:03.973959 6785 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20250112 20:52:03.974602 6785 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55466: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:52:03.974691 6794 ts_tablet_manager.cc:2109] Sending abort request for transaction 1
I20250112 20:52:03.975032 6785 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:55466 (ReqId={client: f72232950f5441f1845ad3766a7ed412, seq_no=1, attempt_no=0}) took 1005 ms. Trace:
I20250112 20:52:03.975162 6785 rpcz_store.cc:276] 0112 20:52:02.969174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:02.969237 (+ 63us) service_pool.cc:225] Handling call
0112 20:52:03.974989 (+1005752us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:02.972238 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:52:02.972302 (+ 64us) participant_op.cc:199] PREPARE: finished
0112 20:52:02.972320 (+ 18us) participant_op.cc:212] START. Timestamp: 1736715122972310
0112 20:52:02.972321 (+ 1us) op_driver.cc:348] REPLICATION: starting
0112 20:52:02.972403 (+ 82us) log.cc:844] Serialized 65 byte log entry
0112 20:52:02.972770 (+ 367us) op_driver.cc:464] REPLICATION: finished
0112 20:52:02.972857 (+ 87us) participant_op.cc:265] APPLY: starting
0112 20:52:02.973044 (+ 187us) participant_op.cc:270] APPLY: finished
0112 20:52:02.973067 (+ 23us) log.cc:844] Serialized 14 byte log entry
0112 20:52:02.973101 (+ 34us) participant_op.cc:94] Released txn lock
0112 20:52:02.973108 (+ 7us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:52:02.973337 (+ 0us) write_op.cc:183] PREPARE: starting on tablet adc3c3b0046c4528b7b3fd2962478478
0112 20:52:02.973404 (+ 67us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:02.973406 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:02.973407 (+ 1us) tablet.cc:659] Decoding operations
0112 20:52:02.973455 (+ 48us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:03.974148 (+1000693us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:52:03.974162 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:03.974168 (+ 6us) write_op.cc:454] Released schema lock
0112 20:52:03.974174 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":48,"txn-participant-registration.run_cpu_time_us":463,"txn-participant-registration.run_wall_time_us":2664,"child_traces":[["op",{"apply.queue_time_us":44,"apply.run_cpu_time_us":469,"apply.run_wall_time_us":468,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":205,"prepare.run_wall_time_us":206,"replication_time_us":432,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":36,"thread_start_us":207,"threads_started":1,"txn-participant-registration.queue_time_us":313}]]}]]}
W20250112 20:52:03.976783 6753 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet adc3c3b0046c4528b7b3fd2962478478 after 1 attempt(s): Failed to write to server: 794ff08a1c4249649eb1c93b804b6aae (127.0.104.2:34777): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:52:03.977097 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:03.986498 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:03.986845 416 tablet_replica.cc:331] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: stopping tablet replica
I20250112 20:52:03.987104 416 raft_consensus.cc:2238] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:03.987376 416 raft_consensus.cc:2267] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:52:03.993275 6785 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations [suppressed 1 similar messages]
I20250112 20:52:04.038429 416 txn_status_manager.cc:765] Waiting for 1 task(s) to stop
I20250112 20:52:04.042424 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:07.047220 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:07.050078 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:07.050347 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:07.050547 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:07.052266 416 server_base.cc:1034] running on GCE node
I20250112 20:52:07.052645 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:07.052731 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:07.052767 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715127052754 us; error 0 us; skew 500 ppm
I20250112 20:52:07.052906 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:07.059644 416 webserver.cc:458] Webserver started at http://127.0.104.1:38667/ using document root <none> and password file <none>
I20250112 20:52:07.059929 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:07.060015 416 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250112 20:52:07.062083 416 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20250112 20:52:07.063597 6813 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:07.063995 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250112 20:52:07.064090 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "7c904621a4004a318d1780673619c909"
format_stamp: "Formatted at 2025-01-12 20:52:01 on dist-test-slave-3gql"
I20250112 20:52:07.064221 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:07.083112 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:07.083777 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:07.086344 6822 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250112 20:52:07.089408 416 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250112 20:52:07.089550 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20250112 20:52:07.089702 416 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250112 20:52:07.092146 416 ts_tablet_manager.cc:610] Registered 1 tablets
I20250112 20:52:07.092245 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250112 20:52:07.092319 6822 tablet_bootstrap.cc:492] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Bootstrap starting.
I20250112 20:52:07.110791 6822 tablet_bootstrap.cc:492] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=4 ignored=0} mutations{seen=1 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250112 20:52:07.111670 6822 tablet_bootstrap.cc:492] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Bootstrap complete.
I20250112 20:52:07.112056 6822 ts_tablet_manager.cc:1397] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20250112 20:52:07.113301 6822 raft_consensus.cc:357] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:07.113654 6822 raft_consensus.cc:738] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c904621a4004a318d1780673619c909, State: Initialized, Role: FOLLOWER
I20250112 20:52:07.114051 6822 consensus_queue.cc:260] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:07.114329 6822 raft_consensus.cc:397] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:07.114435 6822 raft_consensus.cc:491] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:07.114579 6822 raft_consensus.cc:3054] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 1 FOLLOWER]: Advancing to term 2
I20250112 20:52:07.118163 6822 raft_consensus.cc:513] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:07.118556 6822 leader_election.cc:304] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c904621a4004a318d1780673619c909; no voters:
I20250112 20:52:07.119138 6822 leader_election.cc:290] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250112 20:52:07.119324 6867 raft_consensus.cc:2798] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 2 FOLLOWER]: Leader election won for term 2
I20250112 20:52:07.120296 6822 ts_tablet_manager.cc:1428] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20250112 20:52:07.120528 6867 raft_consensus.cc:695] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 2 LEADER]: Becoming Leader. State: Replica: 7c904621a4004a318d1780673619c909, State: Running, Role: LEADER
I20250112 20:52:07.120899 6867 consensus_queue.cc:237] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } }
I20250112 20:52:07.121963 6870 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7c904621a4004a318d1780673619c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } } }
I20250112 20:52:07.122020 6872 tablet_replica.cc:440] TxnStatusTablet state changed. Reason: New leader 7c904621a4004a318d1780673619c909. Latest consensus state: current_term: 2 leader_uuid: "7c904621a4004a318d1780673619c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } } }
I20250112 20:52:07.122272 6870 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:07.122277 6872 tablet_replica.cc:443] This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:07.123183 6878 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:07.123335 6878 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:07.126660 6878 txn_status_manager.cc:728] Starting 1 aborts task
I20250112 20:52:07.129426 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40715
I20250112 20:52:07.131736 6894 heartbeater.cc:346] Connected to a master server at 127.0.104.62:46581
I20250112 20:52:07.131922 6894 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:07.132474 6894 heartbeater.cc:510] Master 127.0.104.62:46581 requested a full tablet report, sending...
I20250112 20:52:07.133827 6520 ts_manager.cc:194] Re-registered known tserver with Master: 7c904621a4004a318d1780673619c909 (127.0.104.1:40715)
I20250112 20:52:07.134239 6520 catalog_manager.cc:5526] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "7c904621a4004a318d1780673619c909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c904621a4004a318d1780673619c909" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40715 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:07.129451 6892 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40715 every 8 connection(s)
I20250112 20:52:07.140841 6520 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48828
I20250112 20:52:07.142274 6894 heartbeater.cc:502] Master 127.0.104.62:46581 was elected leader, sending a full tablet report...
W20250112 20:52:07.148221 6902 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46234: Illegal state: Transaction metadata for transaction 1 already set
I20250112 20:52:07.155236 416 tablet_server.cc:178] TabletServer@127.0.104.1:40715 shutting down...
I20250112 20:52:07.163120 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:07.163442 416 tablet_replica.cc:331] stopping tablet replica
I20250112 20:52:07.163688 416 raft_consensus.cc:2238] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 2 LEADER]: Raft consensus shutting down.
I20250112 20:52:07.163911 416 raft_consensus.cc:2267] T f425f21d85344434b50921b41b60dc84 P 7c904621a4004a318d1780673619c909 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:07.178721 416 tablet_server.cc:195] TabletServer@127.0.104.1:40715 shutdown complete.
I20250112 20:52:07.182986 416 tablet_server.cc:178] TabletServer@127.0.104.2:0 shutting down...
I20250112 20:52:07.191625 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:07.191905 416 tablet_replica.cc:331] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae: stopping tablet replica
I20250112 20:52:07.192107 416 raft_consensus.cc:2238] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:07.192281 416 raft_consensus.cc:2267] T adc3c3b0046c4528b7b3fd2962478478 P 794ff08a1c4249649eb1c93b804b6aae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:07.206308 416 tablet_server.cc:195] TabletServer@127.0.104.2:0 shutdown complete.
I20250112 20:52:07.209968 416 master.cc:537] Master@127.0.104.62:46581 shutting down...
I20250112 20:52:07.218582 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:07.218909 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:07.219079 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 62a56af2da7b4bff9e4dbdd101d5d080: stopping tablet replica
I20250112 20:52:07.233156 416 master.cc:559] Master@127.0.104.62:46581 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestRetryWaitDieAbortsWhenTServerUnavailable (5674 ms)
[ RUN ] TxnOpDispatcherITest.BeginTxnLockAbort
I20250112 20:52:07.244053 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:42863
I20250112 20:52:07.244657 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:07.246909 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:07.247113 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:07.247509 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:07.248276 416 server_base.cc:1034] running on GCE node
I20250112 20:52:07.248567 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:07.248664 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:07.248754 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715127248702 us; error 0 us; skew 500 ppm
I20250112 20:52:07.248965 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:07.250217 416 webserver.cc:458] Webserver started at http://127.0.104.62:39105/ using document root <none> and password file <none>
I20250112 20:52:07.250456 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:07.250535 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:07.250702 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:07.251348 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "c3c1656c365745259fb5d51561ca701a"
format_stamp: "Formatted at 2025-01-12 20:52:07 on dist-test-slave-3gql"
I20250112 20:52:07.254002 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:07.255976 6913 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:07.256338 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250112 20:52:07.256469 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "c3c1656c365745259fb5d51561ca701a"
format_stamp: "Formatted at 2025-01-12 20:52:07 on dist-test-slave-3gql"
I20250112 20:52:07.256652 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:07.271636 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:07.272251 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:07.288436 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42863
I20250112 20:52:07.288455 6974 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42863 every 8 connection(s)
I20250112 20:52:07.289837 6975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:07.293668 6975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a: Bootstrap starting.
I20250112 20:52:07.295053 6975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:07.296636 6975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a: No bootstrap required, opened a new log
I20250112 20:52:07.297250 6975 raft_consensus.cc:357] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1656c365745259fb5d51561ca701a" member_type: VOTER }
I20250112 20:52:07.297387 6975 raft_consensus.cc:383] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:07.297456 6975 raft_consensus.cc:738] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3c1656c365745259fb5d51561ca701a, State: Initialized, Role: FOLLOWER
I20250112 20:52:07.297672 6975 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1656c365745259fb5d51561ca701a" member_type: VOTER }
I20250112 20:52:07.297809 6975 raft_consensus.cc:397] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:07.297885 6975 raft_consensus.cc:491] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:07.297987 6975 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:07.299515 6975 raft_consensus.cc:513] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1656c365745259fb5d51561ca701a" member_type: VOTER }
I20250112 20:52:07.299712 6975 leader_election.cc:304] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3c1656c365745259fb5d51561ca701a; no voters:
I20250112 20:52:07.300098 6975 leader_election.cc:290] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:07.300264 6978 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:07.300779 6978 raft_consensus.cc:695] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 1 LEADER]: Becoming Leader. State: Replica: c3c1656c365745259fb5d51561ca701a, State: Running, Role: LEADER
I20250112 20:52:07.301054 6975 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:07.301071 6978 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1656c365745259fb5d51561ca701a" member_type: VOTER }
I20250112 20:52:07.302134 6979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3c1656c365745259fb5d51561ca701a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1656c365745259fb5d51561ca701a" member_type: VOTER } }
I20250112 20:52:07.302186 6980 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3c1656c365745259fb5d51561ca701a. Latest consensus state: current_term: 1 leader_uuid: "c3c1656c365745259fb5d51561ca701a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3c1656c365745259fb5d51561ca701a" member_type: VOTER } }
I20250112 20:52:07.302462 6980 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:07.302989 6979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:07.305168 6983 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:07.306700 6983 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:07.307077 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:07.310518 6983 catalog_manager.cc:1348] Generated new cluster ID: 8357814b6d9341b193592907b64c1a0a
I20250112 20:52:07.310635 6983 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:07.329734 6983 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:07.330503 6983 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:07.339612 6983 catalog_manager.cc:5899] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a: Generated new TSK 0
I20250112 20:52:07.339851 6983 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:07.351114 6930 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33738:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:07.372340 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:07.375269 7003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:07.375300 7002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:07.375849 7005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:07.376335 416 server_base.cc:1034] running on GCE node
I20250112 20:52:07.376637 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:07.376710 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:07.376816 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715127376776 us; error 0 us; skew 500 ppm
I20250112 20:52:07.376992 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:07.378348 416 webserver.cc:458] Webserver started at http://127.0.104.1:39773/ using document root <none> and password file <none>
I20250112 20:52:07.378593 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:07.378705 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:07.378844 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:07.379516 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "35e8349935284b54b5aa7c39cf7627c8"
format_stamp: "Formatted at 2025-01-12 20:52:07 on dist-test-slave-3gql"
I20250112 20:52:07.382155 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250112 20:52:07.383749 7010 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:07.384114 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:07.384234 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "35e8349935284b54b5aa7c39cf7627c8"
format_stamp: "Formatted at 2025-01-12 20:52:07 on dist-test-slave-3gql"
I20250112 20:52:07.384372 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockAbort.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:07.390897 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:07.391495 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:07.393338 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:07.393443 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:07.393559 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:07.393649 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:07.412537 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46791
I20250112 20:52:07.412622 7079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46791 every 8 connection(s)
I20250112 20:52:07.414553 7080 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42863
I20250112 20:52:07.414695 7080 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:07.415107 7080 heartbeater.cc:510] Master 127.0.104.62:42863 requested a full tablet report, sending...
I20250112 20:52:07.415946 6930 ts_manager.cc:194] Registered new tserver with Master: 35e8349935284b54b5aa7c39cf7627c8 (127.0.104.1:46791)
I20250112 20:52:07.416149 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002363351s
I20250112 20:52:07.417191 6930 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33754
I20250112 20:52:07.427425 6930 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33786:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:07.440739 7044 tablet_service.cc:1467] Processing CreateTablet for tablet a7e56fe20af14f0abe4dd452a74e09eb (DEFAULT_TABLE table=txn_op_dispatcher_test [id=2ecee7f43cda404aacdb509ebc152652]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:07.441272 7044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e56fe20af14f0abe4dd452a74e09eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:07.441305 7042 tablet_service.cc:1467] Processing CreateTablet for tablet 6741957016c74bdeb19a263eb8694ad1 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=2ecee7f43cda404aacdb509ebc152652]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:07.441758 7042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6741957016c74bdeb19a263eb8694ad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:07.446374 7090 tablet_bootstrap.cc:492] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8: Bootstrap starting.
I20250112 20:52:07.447850 7090 tablet_bootstrap.cc:654] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:07.449781 7090 tablet_bootstrap.cc:492] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8: No bootstrap required, opened a new log
I20250112 20:52:07.449923 7090 ts_tablet_manager.cc:1397] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:52:07.450517 7090 raft_consensus.cc:357] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.450671 7090 raft_consensus.cc:383] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:07.450775 7090 raft_consensus.cc:738] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e8349935284b54b5aa7c39cf7627c8, State: Initialized, Role: FOLLOWER
I20250112 20:52:07.451014 7090 consensus_queue.cc:260] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.451174 7090 raft_consensus.cc:397] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:07.451242 7090 raft_consensus.cc:491] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:07.451359 7090 raft_consensus.cc:3054] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:07.453155 7090 raft_consensus.cc:513] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.453425 7090 leader_election.cc:304] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e8349935284b54b5aa7c39cf7627c8; no voters:
I20250112 20:52:07.454005 7090 leader_election.cc:290] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:07.454157 7092 raft_consensus.cc:2798] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:07.454710 7080 heartbeater.cc:502] Master 127.0.104.62:42863 was elected leader, sending a full tablet report...
I20250112 20:52:07.454705 7092 raft_consensus.cc:695] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 1 LEADER]: Becoming Leader. State: Replica: 35e8349935284b54b5aa7c39cf7627c8, State: Running, Role: LEADER
I20250112 20:52:07.455097 7090 ts_tablet_manager.cc:1428] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250112 20:52:07.455369 7090 tablet_bootstrap.cc:492] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8: Bootstrap starting.
I20250112 20:52:07.455343 7092 consensus_queue.cc:237] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.457119 7090 tablet_bootstrap.cc:654] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:07.458115 6930 catalog_manager.cc:5526] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e8349935284b54b5aa7c39cf7627c8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "35e8349935284b54b5aa7c39cf7627c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:07.459187 7090 tablet_bootstrap.cc:492] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8: No bootstrap required, opened a new log
I20250112 20:52:07.459406 7090 ts_tablet_manager.cc:1397] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:07.460258 7090 raft_consensus.cc:357] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.460441 7090 raft_consensus.cc:383] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:07.460539 7090 raft_consensus.cc:738] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e8349935284b54b5aa7c39cf7627c8, State: Initialized, Role: FOLLOWER
I20250112 20:52:07.460824 7090 consensus_queue.cc:260] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.461041 7090 raft_consensus.cc:397] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:07.461153 7090 raft_consensus.cc:491] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:07.461256 7090 raft_consensus.cc:3054] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:07.463662 7090 raft_consensus.cc:513] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.463866 7090 leader_election.cc:304] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e8349935284b54b5aa7c39cf7627c8; no voters:
I20250112 20:52:07.464061 7090 leader_election.cc:290] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:07.464121 7092 raft_consensus.cc:2798] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:07.464309 7092 raft_consensus.cc:695] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 1 LEADER]: Becoming Leader. State: Replica: 35e8349935284b54b5aa7c39cf7627c8, State: Running, Role: LEADER
I20250112 20:52:07.464488 7090 ts_tablet_manager.cc:1428] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250112 20:52:07.464536 7092 consensus_queue.cc:237] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:07.466395 6930 catalog_manager.cc:5526] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e8349935284b54b5aa7c39cf7627c8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "35e8349935284b54b5aa7c39cf7627c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:07.478538 7083 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:08.366560 6930 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33796:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:08.375144 7044 tablet_service.cc:1467] Processing CreateTablet for tablet da145d4464754f9e8dc3dbf8e4cc6eab (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=95b4414e57d64526b33c2f627a3f1cb8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:08.375586 7044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da145d4464754f9e8dc3dbf8e4cc6eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:08.379727 7108 tablet_bootstrap.cc:492] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: Bootstrap starting.
I20250112 20:52:08.381364 7108 tablet_bootstrap.cc:654] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:08.382912 7108 tablet_bootstrap.cc:492] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: No bootstrap required, opened a new log
I20250112 20:52:08.383057 7108 ts_tablet_manager.cc:1397] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20250112 20:52:08.383757 7108 raft_consensus.cc:357] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:08.383942 7108 raft_consensus.cc:383] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:08.384056 7108 raft_consensus.cc:738] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e8349935284b54b5aa7c39cf7627c8, State: Initialized, Role: FOLLOWER
I20250112 20:52:08.384266 7108 consensus_queue.cc:260] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:08.384419 7108 raft_consensus.cc:397] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:08.384490 7108 raft_consensus.cc:491] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:08.384630 7108 raft_consensus.cc:3054] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:08.386305 7108 raft_consensus.cc:513] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:08.386519 7108 leader_election.cc:304] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35e8349935284b54b5aa7c39cf7627c8; no voters:
I20250112 20:52:08.386969 7108 leader_election.cc:290] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:08.387110 7110 raft_consensus.cc:2798] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:08.387658 7108 ts_tablet_manager.cc:1428] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20250112 20:52:08.388201 7110 raft_consensus.cc:695] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 1 LEADER]: Becoming Leader. State: Replica: 35e8349935284b54b5aa7c39cf7627c8, State: Running, Role: LEADER
I20250112 20:52:08.388456 7110 consensus_queue.cc:237] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } }
I20250112 20:52:08.389472 7111 tablet_replica.cc:440] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35e8349935284b54b5aa7c39cf7627c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } } }
I20250112 20:52:08.389704 7111 tablet_replica.cc:443] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:08.389577 7112 tablet_replica.cc:440] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: TxnStatusTablet state changed. Reason: New leader 35e8349935284b54b5aa7c39cf7627c8. Latest consensus state: current_term: 1 leader_uuid: "35e8349935284b54b5aa7c39cf7627c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } } }
I20250112 20:52:08.389811 7112 tablet_replica.cc:443] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:08.390846 6930 catalog_manager.cc:5526] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35e8349935284b54b5aa7c39cf7627c8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "35e8349935284b54b5aa7c39cf7627c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35e8349935284b54b5aa7c39cf7627c8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46791 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:08.396764 7114 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:08.396943 7114 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250112 20:52:09.558825 7115 lock_manager.cc:494] Txn 9223372036854775807 has not acquired the partition lock after 1000ms
W20250112 20:52:09.559105 7115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43202: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:52:09.559408 7115 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:43202 (ReqId={client: 5c309a70465443d8855defeacde48009, seq_no=6, attempt_no=0}) took 1001 ms. Trace:
I20250112 20:52:09.559499 7115 rpcz_store.cc:276] 0112 20:52:08.557768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:08.557819 (+ 51us) service_pool.cc:225] Handling call
0112 20:52:09.559361 (+1001542us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:08.558205 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 6741957016c74bdeb19a263eb8694ad1
0112 20:52:08.558263 (+ 58us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:08.558267 (+ 4us) write_op.cc:435] Acquired schema lock
0112 20:52:08.558268 (+ 1us) tablet.cc:659] Decoding operations
0112 20:52:08.558313 (+ 45us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:09.559050 (+1000737us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:52:09.559066 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:09.559074 (+ 8us) write_op.cc:454] Released schema lock
0112 20:52:09.559081 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":47}]]}
W20250112 20:52:09.560185 7083 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 6741957016c74bdeb19a263eb8694ad1 after 1 attempt(s): Failed to write to server: 35e8349935284b54b5aa7c39cf7627c8 (127.0.104.1:46791): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:52:10.572301 7115 lock_manager.cc:494] Txn 1 has not acquired the partition lock after 1000ms
W20250112 20:52:10.572850 7115 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43202: Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
I20250112 20:52:10.572971 7131 ts_tablet_manager.cc:2109] Sending abort request for transaction 1
I20250112 20:52:10.573171 7115 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:43202 (ReqId={client: 5c309a70465443d8855defeacde48009, seq_no=7, attempt_no=0}) took 1006 ms. Trace:
I20250112 20:52:10.573256 7115 rpcz_store.cc:276] 0112 20:52:09.566477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:09.566536 (+ 59us) service_pool.cc:225] Handling call
0112 20:52:10.573110 (+1006574us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:09.570207 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:52:09.570238 (+ 31us) participant_op.cc:199] PREPARE: finished
0112 20:52:09.570260 (+ 22us) participant_op.cc:212] START. Timestamp: 1736715129570248
0112 20:52:09.570262 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:52:09.570356 (+ 94us) log.cc:844] Serialized 65 byte log entry
0112 20:52:09.571105 (+ 749us) op_driver.cc:464] REPLICATION: finished
0112 20:52:09.571188 (+ 83us) participant_op.cc:265] APPLY: starting
0112 20:52:09.571385 (+ 197us) participant_op.cc:270] APPLY: finished
0112 20:52:09.571410 (+ 25us) log.cc:844] Serialized 14 byte log entry
0112 20:52:09.571441 (+ 31us) participant_op.cc:94] Released txn lock
0112 20:52:09.571446 (+ 5us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:52:09.571677 (+ 0us) write_op.cc:183] PREPARE: starting on tablet a7e56fe20af14f0abe4dd452a74e09eb
0112 20:52:09.571762 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:09.571764 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:09.571765 (+ 1us) tablet.cc:659] Decoding operations
0112 20:52:09.571823 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:10.572477 (+1000654us) op_driver.cc:389] HandleFailure(Aborted: Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID)
0112 20:52:10.572494 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:10.572504 (+ 10us) write_op.cc:454] Released schema lock
0112 20:52:10.572510 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"thread_start_us":187,"threads_started":1,"txn-participant-registration.queue_time_us":273,"txn-participant-registration.run_cpu_time_us":554,"txn-participant-registration.run_wall_time_us":3083,"child_traces":[["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":463,"apply.run_wall_time_us":463,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":394,"prepare.run_wall_time_us":394,"raft.queue_time_us":69,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"replication_time_us":826,"thread_start_us":188,"threads_started":1,"wal-append.queue_time_us":260,"wal-append.run_cpu_time_us":600,"wal-append.run_wall_time_us":1001055,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":36,"thread_start_us":206,"threads_started":1,"txn-participant-registration.queue_time_us":323}]]}]]}
W20250112 20:52:10.574256 7083 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet a7e56fe20af14f0abe4dd452a74e09eb after 1 attempt(s): Failed to write to server: 35e8349935284b54b5aa7c39cf7627c8 (127.0.104.1:46791): Write op should be aborted since it tries to acquire the partition lock that is held by another transaction that has lower txn ID
W20250112 20:52:10.579505 7115 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
W20250112 20:52:10.601500 7131 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:43202: Illegal state: Failed to write to server: 35e8349935284b54b5aa7c39cf7627c8 (127.0.104.1:46791): transaction ID 1 not open: ABORTED
I20250112 20:52:10.604321 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:10.612180 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:10.612565 416 tablet_replica.cc:331] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8: stopping tablet replica
I20250112 20:52:10.612857 416 raft_consensus.cc:2238] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:10.613044 416 raft_consensus.cc:2267] T da145d4464754f9e8dc3dbf8e4cc6eab P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:10.613762 416 tablet_replica.cc:331] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8: stopping tablet replica
I20250112 20:52:10.613881 416 raft_consensus.cc:2238] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:10.614032 416 raft_consensus.cc:2267] T a7e56fe20af14f0abe4dd452a74e09eb P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:10.614564 416 tablet_replica.cc:331] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8: stopping tablet replica
I20250112 20:52:10.614693 416 raft_consensus.cc:2238] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:10.614811 416 raft_consensus.cc:2267] T 6741957016c74bdeb19a263eb8694ad1 P 35e8349935284b54b5aa7c39cf7627c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:10.629071 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:10.633005 416 master.cc:537] Master@127.0.104.62:42863 shutting down...
I20250112 20:52:10.640955 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:10.641196 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:10.641317 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P c3c1656c365745259fb5d51561ca701a: stopping tablet replica
I20250112 20:52:10.654465 416 master.cc:559] Master@127.0.104.62:42863 shutdown complete.
[ OK ] TxnOpDispatcherITest.BeginTxnLockAbort (3420 ms)
[ RUN ] TxnOpDispatcherITest.BeginTxnLockRetry
I20250112 20:52:10.665128 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:40113
I20250112 20:52:10.665655 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:10.667950 7140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:10.668087 7141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:10.668406 7143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:10.668744 416 server_base.cc:1034] running on GCE node
I20250112 20:52:10.669034 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:10.669109 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:10.669152 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715130669137 us; error 0 us; skew 500 ppm
I20250112 20:52:10.669288 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:10.670480 416 webserver.cc:458] Webserver started at http://127.0.104.62:40093/ using document root <none> and password file <none>
I20250112 20:52:10.670694 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:10.670761 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:10.670877 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:10.671500 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "2a77ee99523041d9aa8bb369eac61f96"
format_stamp: "Formatted at 2025-01-12 20:52:10 on dist-test-slave-3gql"
I20250112 20:52:10.673903 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:10.675478 7148 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:10.675794 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:10.675917 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "2a77ee99523041d9aa8bb369eac61f96"
format_stamp: "Formatted at 2025-01-12 20:52:10 on dist-test-slave-3gql"
I20250112 20:52:10.676048 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:10.687265 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:10.687894 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:10.702586 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40113
I20250112 20:52:10.702625 7209 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40113 every 8 connection(s)
I20250112 20:52:10.704322 7210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:10.710322 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96: Bootstrap starting.
I20250112 20:52:10.712180 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:10.713889 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96: No bootstrap required, opened a new log
I20250112 20:52:10.714550 7210 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a77ee99523041d9aa8bb369eac61f96" member_type: VOTER }
I20250112 20:52:10.714699 7210 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:10.714792 7210 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a77ee99523041d9aa8bb369eac61f96, State: Initialized, Role: FOLLOWER
I20250112 20:52:10.715015 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a77ee99523041d9aa8bb369eac61f96" member_type: VOTER }
I20250112 20:52:10.715162 7210 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:10.715238 7210 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:10.715351 7210 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:10.716958 7210 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a77ee99523041d9aa8bb369eac61f96" member_type: VOTER }
I20250112 20:52:10.717160 7210 leader_election.cc:304] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a77ee99523041d9aa8bb369eac61f96; no voters:
I20250112 20:52:10.717559 7210 leader_election.cc:290] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:10.717712 7213 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:10.718161 7213 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 1 LEADER]: Becoming Leader. State: Replica: 2a77ee99523041d9aa8bb369eac61f96, State: Running, Role: LEADER
I20250112 20:52:10.718469 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:10.718448 7213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a77ee99523041d9aa8bb369eac61f96" member_type: VOTER }
I20250112 20:52:10.719146 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a77ee99523041d9aa8bb369eac61f96. Latest consensus state: current_term: 1 leader_uuid: "2a77ee99523041d9aa8bb369eac61f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a77ee99523041d9aa8bb369eac61f96" member_type: VOTER } }
I20250112 20:52:10.719292 7214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a77ee99523041d9aa8bb369eac61f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a77ee99523041d9aa8bb369eac61f96" member_type: VOTER } }
I20250112 20:52:10.719419 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:10.719446 7214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:10.720954 7222 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:10.723042 7222 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:10.723407 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:10.726351 7222 catalog_manager.cc:1348] Generated new cluster ID: 096dc70aa2514c6abd61b0e6a4eff2db
I20250112 20:52:10.726461 7222 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:10.742857 7222 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:10.743777 7222 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:10.751488 7222 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96: Generated new TSK 0
I20250112 20:52:10.751740 7222 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:10.756011 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:10.758786 7237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:10.759496 7238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:10.760219 7240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:10.760932 416 server_base.cc:1034] running on GCE node
I20250112 20:52:10.761227 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:10.761296 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:10.761382 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715130761342 us; error 0 us; skew 500 ppm
I20250112 20:52:10.761559 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:10.762413 7165 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33358:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:10.763163 416 webserver.cc:458] Webserver started at http://127.0.104.1:41953/ using document root <none> and password file <none>
I20250112 20:52:10.763442 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:10.763573 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:10.763749 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:10.764434 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "0282d3d0c17f42199a78da799f1a5c01"
format_stamp: "Formatted at 2025-01-12 20:52:10 on dist-test-slave-3gql"
I20250112 20:52:10.766993 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250112 20:52:10.768651 7245 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:10.769124 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:10.769268 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "0282d3d0c17f42199a78da799f1a5c01"
format_stamp: "Formatted at 2025-01-12 20:52:10 on dist-test-slave-3gql"
I20250112 20:52:10.769488 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.BeginTxnLockRetry.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:10.778863 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:10.779412 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:10.781200 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:10.781380 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:10.781481 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:10.781598 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:10.803166 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40841
I20250112 20:52:10.803208 7314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40841 every 8 connection(s)
I20250112 20:52:10.805768 7315 heartbeater.cc:346] Connected to a master server at 127.0.104.62:40113
I20250112 20:52:10.805909 7315 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:10.806291 7315 heartbeater.cc:510] Master 127.0.104.62:40113 requested a full tablet report, sending...
I20250112 20:52:10.807054 7165 ts_manager.cc:194] Registered new tserver with Master: 0282d3d0c17f42199a78da799f1a5c01 (127.0.104.1:40841)
I20250112 20:52:10.807457 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00336239s
I20250112 20:52:10.808138 7165 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33374
I20250112 20:52:10.818429 7165 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33384:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:10.831765 7280 tablet_service.cc:1467] Processing CreateTablet for tablet 3604da306304470285a571d6e4e9ae28 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=02fb3604245242eca03690a8f801f1e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:10.831806 7278 tablet_service.cc:1467] Processing CreateTablet for tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd (DEFAULT_TABLE table=txn_op_dispatcher_test [id=02fb3604245242eca03690a8f801f1e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:10.832245 7280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3604da306304470285a571d6e4e9ae28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:10.832463 7278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:10.837147 7325 tablet_bootstrap.cc:492] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01: Bootstrap starting.
I20250112 20:52:10.838582 7325 tablet_bootstrap.cc:654] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:10.840267 7325 tablet_bootstrap.cc:492] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01: No bootstrap required, opened a new log
I20250112 20:52:10.840396 7325 ts_tablet_manager.cc:1397] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:52:10.841040 7325 raft_consensus.cc:357] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.841192 7325 raft_consensus.cc:383] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:10.841285 7325 raft_consensus.cc:738] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0282d3d0c17f42199a78da799f1a5c01, State: Initialized, Role: FOLLOWER
I20250112 20:52:10.841490 7325 consensus_queue.cc:260] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.841650 7325 raft_consensus.cc:397] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:10.841718 7325 raft_consensus.cc:491] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:10.841838 7325 raft_consensus.cc:3054] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:10.843482 7325 raft_consensus.cc:513] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.843724 7325 leader_election.cc:304] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0282d3d0c17f42199a78da799f1a5c01; no voters:
I20250112 20:52:10.844182 7325 leader_election.cc:290] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:10.844333 7327 raft_consensus.cc:2798] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:10.844887 7327 raft_consensus.cc:695] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 1 LEADER]: Becoming Leader. State: Replica: 0282d3d0c17f42199a78da799f1a5c01, State: Running, Role: LEADER
I20250112 20:52:10.844950 7315 heartbeater.cc:502] Master 127.0.104.62:40113 was elected leader, sending a full tablet report...
I20250112 20:52:10.844977 7325 ts_tablet_manager.cc:1428] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:10.845397 7325 tablet_bootstrap.cc:492] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01: Bootstrap starting.
I20250112 20:52:10.845326 7327 consensus_queue.cc:237] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.847424 7325 tablet_bootstrap.cc:654] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:10.847729 7165 catalog_manager.cc:5526] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0282d3d0c17f42199a78da799f1a5c01 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "0282d3d0c17f42199a78da799f1a5c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:10.849205 7325 tablet_bootstrap.cc:492] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01: No bootstrap required, opened a new log
I20250112 20:52:10.849370 7325 ts_tablet_manager.cc:1397] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250112 20:52:10.850194 7325 raft_consensus.cc:357] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.850417 7325 raft_consensus.cc:383] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:10.850504 7325 raft_consensus.cc:738] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0282d3d0c17f42199a78da799f1a5c01, State: Initialized, Role: FOLLOWER
I20250112 20:52:10.850744 7325 consensus_queue.cc:260] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.850885 7325 raft_consensus.cc:397] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:10.850982 7325 raft_consensus.cc:491] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:10.851085 7325 raft_consensus.cc:3054] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:10.852905 7325 raft_consensus.cc:513] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.853096 7325 leader_election.cc:304] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0282d3d0c17f42199a78da799f1a5c01; no voters:
I20250112 20:52:10.853267 7325 leader_election.cc:290] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:10.853325 7327 raft_consensus.cc:2798] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:10.853536 7327 raft_consensus.cc:695] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 LEADER]: Becoming Leader. State: Replica: 0282d3d0c17f42199a78da799f1a5c01, State: Running, Role: LEADER
I20250112 20:52:10.853646 7325 ts_tablet_manager.cc:1428] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20250112 20:52:10.853775 7327 consensus_queue.cc:237] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:10.855472 7165 catalog_manager.cc:5526] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0282d3d0c17f42199a78da799f1a5c01 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "0282d3d0c17f42199a78da799f1a5c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:10.867996 7319 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:11.775327 7165 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33400:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:11.782637 7278 tablet_service.cc:1467] Processing CreateTablet for tablet 81c45183bcd848e6a249675d7961e0a1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15d55b994795421b92304ac186d42738]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:11.783106 7278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c45183bcd848e6a249675d7961e0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:11.787392 7343 tablet_bootstrap.cc:492] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: Bootstrap starting.
I20250112 20:52:11.789086 7343 tablet_bootstrap.cc:654] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:11.790573 7343 tablet_bootstrap.cc:492] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: No bootstrap required, opened a new log
I20250112 20:52:11.790688 7343 ts_tablet_manager.cc:1397] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:52:11.791399 7343 raft_consensus.cc:357] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:11.791572 7343 raft_consensus.cc:383] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:11.791718 7343 raft_consensus.cc:738] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0282d3d0c17f42199a78da799f1a5c01, State: Initialized, Role: FOLLOWER
I20250112 20:52:11.791957 7343 consensus_queue.cc:260] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:11.792160 7343 raft_consensus.cc:397] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:11.792251 7343 raft_consensus.cc:491] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:11.792359 7343 raft_consensus.cc:3054] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:11.794034 7343 raft_consensus.cc:513] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:11.794250 7343 leader_election.cc:304] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0282d3d0c17f42199a78da799f1a5c01; no voters:
I20250112 20:52:11.794642 7343 leader_election.cc:290] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:11.794787 7345 raft_consensus.cc:2798] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:11.795363 7343 ts_tablet_manager.cc:1428] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20250112 20:52:11.795415 7345 raft_consensus.cc:695] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 LEADER]: Becoming Leader. State: Replica: 0282d3d0c17f42199a78da799f1a5c01, State: Running, Role: LEADER
I20250112 20:52:11.795735 7345 consensus_queue.cc:237] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } }
I20250112 20:52:11.796792 7347 tablet_replica.cc:440] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: TxnStatusTablet state changed. Reason: New leader 0282d3d0c17f42199a78da799f1a5c01. Latest consensus state: current_term: 1 leader_uuid: "0282d3d0c17f42199a78da799f1a5c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } } }
I20250112 20:52:11.796855 7346 tablet_replica.cc:440] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0282d3d0c17f42199a78da799f1a5c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } } }
I20250112 20:52:11.797052 7347 tablet_replica.cc:443] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:11.797263 7346 tablet_replica.cc:443] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:11.798250 7165 catalog_manager.cc:5526] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0282d3d0c17f42199a78da799f1a5c01 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "0282d3d0c17f42199a78da799f1a5c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0282d3d0c17f42199a78da799f1a5c01" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40841 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:11.799677 7350 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:11.799836 7350 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250112 20:52:13.503062 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:13.503332 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:13.503695 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=0}) took 1005 ms. Trace:
I20250112 20:52:13.503793 7354 rpcz_store.cc:276] 0112 20:52:12.498299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:12.498351 (+ 52us) service_pool.cc:225] Handling call
0112 20:52:13.503616 (+1005265us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:12.501301 (+ 0us) participant_op.cc:163] PREPARE: starting
0112 20:52:12.501329 (+ 28us) participant_op.cc:199] PREPARE: finished
0112 20:52:12.501346 (+ 17us) participant_op.cc:212] START. Timestamp: 1736715132501337
0112 20:52:12.501348 (+ 2us) op_driver.cc:348] REPLICATION: starting
0112 20:52:12.501442 (+ 94us) log.cc:844] Serialized 65 byte log entry
0112 20:52:12.501785 (+ 343us) op_driver.cc:464] REPLICATION: finished
0112 20:52:12.501872 (+ 87us) participant_op.cc:265] APPLY: starting
0112 20:52:12.502058 (+ 186us) participant_op.cc:270] APPLY: finished
0112 20:52:12.502081 (+ 23us) log.cc:844] Serialized 14 byte log entry
0112 20:52:12.502116 (+ 35us) participant_op.cc:94] Released txn lock
0112 20:52:12.502122 (+ 6us) participant_op.cc:291] FINISH: Op applied
Related trace 'op':
0112 20:52:12.502355 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:12.502418 (+ 63us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:12.502420 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:12.502421 (+ 1us) tablet.cc:659] Decoding operations
0112 20:52:12.502465 (+ 44us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:13.503254 (+1000789us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:13.503279 (+ 25us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:13.503290 (+ 11us) write_op.cc:454] Released schema lock
0112 20:52:13.503300 (+ 10us) write_op.cc:335] FINISH: Op aborted
Metrics: {"txn-participant-registration.queue_time_us":48,"txn-participant-registration.run_cpu_time_us":472,"txn-participant-registration.run_wall_time_us":2658,"child_traces":[["op",{"apply.queue_time_us":44,"apply.run_cpu_time_us":473,"apply.run_wall_time_us":473,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":182,"prepare.run_wall_time_us":183,"replication_time_us":421,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":39}]]}]]}
W20250112 20:52:14.510624 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:14.510917 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:14.511269 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=1}) took 1001 ms. Trace:
I20250112 20:52:14.511370 7354 rpcz_store.cc:276] 0112 20:52:13.509317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:13.509385 (+ 68us) service_pool.cc:225] Handling call
0112 20:52:14.511182 (+1001797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:13.509920 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:13.510047 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:13.510050 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:52:13.510051 (+ 1us) tablet.cc:659] Decoding operations
0112 20:52:13.510131 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:14.510839 (+1000708us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:14.510868 (+ 29us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:14.510877 (+ 9us) write_op.cc:454] Released schema lock
0112 20:52:14.510886 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":74}]]}
W20250112 20:52:15.520325 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:15.520601 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:15.520876 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=2}) took 1002 ms. Trace:
I20250112 20:52:15.520953 7354 rpcz_store.cc:276] 0112 20:52:14.518874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:14.518962 (+ 88us) service_pool.cc:225] Handling call
0112 20:52:15.520827 (+1001865us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:14.519628 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:14.519750 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:14.519752 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:14.519754 (+ 2us) tablet.cc:659] Decoding operations
0112 20:52:14.519829 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:15.520503 (+1000674us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:15.520522 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:15.520533 (+ 11us) write_op.cc:454] Released schema lock
0112 20:52:15.520568 (+ 35us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":98}]]}
W20250112 20:52:16.528872 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:16.529129 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:16.529424 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=3}) took 1002 ms. Trace:
I20250112 20:52:16.529496 7354 rpcz_store.cc:276] 0112 20:52:15.527335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:15.527433 (+ 98us) service_pool.cc:225] Handling call
0112 20:52:16.529379 (+1001946us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:15.528072 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:15.528190 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:15.528192 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:15.528194 (+ 2us) tablet.cc:659] Decoding operations
0112 20:52:15.528265 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:16.529064 (+1000799us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:16.529078 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:16.529088 (+ 10us) write_op.cc:454] Released schema lock
0112 20:52:16.529096 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":64}]]}
W20250112 20:52:17.540575 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:17.540858 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:17.541154 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=4}) took 1002 ms. Trace:
I20250112 20:52:17.541232 7354 rpcz_store.cc:276] 0112 20:52:16.539031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:16.539135 (+ 104us) service_pool.cc:225] Handling call
0112 20:52:17.541095 (+1001960us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:16.539763 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:16.539888 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:16.539890 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:16.539892 (+ 2us) tablet.cc:659] Decoding operations
0112 20:52:16.539968 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:17.540783 (+1000815us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:17.540802 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:17.540811 (+ 9us) write_op.cc:454] Released schema lock
0112 20:52:17.540822 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":93}]]}
W20250112 20:52:18.553366 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:18.553653 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:18.553974 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=5}) took 1002 ms. Trace:
I20250112 20:52:18.554069 7354 rpcz_store.cc:276] 0112 20:52:17.551712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:17.551849 (+ 137us) service_pool.cc:225] Handling call
0112 20:52:18.553915 (+1002066us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:17.552427 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:17.552555 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:17.552557 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:17.552559 (+ 2us) tablet.cc:659] Decoding operations
0112 20:52:17.552677 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:18.553582 (+1000905us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:18.553603 (+ 21us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:18.553614 (+ 11us) write_op.cc:454] Released schema lock
0112 20:52:18.553625 (+ 11us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":66}]]}
W20250112 20:52:19.563050 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:19.563297 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:19.563582 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=6}) took 1002 ms. Trace:
I20250112 20:52:19.563657 7354 rpcz_store.cc:276] 0112 20:52:18.561470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:18.561588 (+ 118us) service_pool.cc:225] Handling call
0112 20:52:19.563533 (+1001945us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:18.562238 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:18.562351 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:18.562353 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:18.562355 (+ 2us) tablet.cc:659] Decoding operations
0112 20:52:18.562428 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:19.563226 (+1000798us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:19.563240 (+ 14us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:19.563250 (+ 10us) write_op.cc:454] Released schema lock
0112 20:52:19.563262 (+ 12us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":68}]]}
W20250112 20:52:20.576566 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:20.576817 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:20.577106 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=7}) took 1002 ms. Trace:
I20250112 20:52:20.577195 7354 rpcz_store.cc:276] 0112 20:52:19.575049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:19.575139 (+ 90us) service_pool.cc:225] Handling call
0112 20:52:20.577061 (+1001922us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:19.575745 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:19.575865 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:19.575867 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:19.575869 (+ 2us) tablet.cc:659] Decoding operations
0112 20:52:19.575946 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:20.576746 (+1000800us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:20.576762 (+ 16us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:20.576770 (+ 8us) write_op.cc:454] Released schema lock
0112 20:52:20.576777 (+ 7us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":65}]]}
W20250112 20:52:21.590996 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:21.591238 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:21.591508 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=8}) took 1002 ms. Trace:
I20250112 20:52:21.591583 7354 rpcz_store.cc:276] 0112 20:52:20.589496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:20.589599 (+ 103us) service_pool.cc:225] Handling call
0112 20:52:21.591459 (+1001860us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:20.590220 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:20.590338 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:20.590341 (+ 3us) write_op.cc:435] Acquired schema lock
0112 20:52:20.590342 (+ 1us) tablet.cc:659] Decoding operations
0112 20:52:20.590418 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:21.591173 (+1000755us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:21.591191 (+ 18us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:21.591202 (+ 11us) write_op.cc:454] Released schema lock
0112 20:52:21.591211 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":81}]]}
W20250112 20:52:22.606650 7354 lock_manager.cc:494] Txn 0 has not acquired the partition lock after 1000ms
W20250112 20:52:22.606904 7354 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39528: Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID
I20250112 20:52:22.607203 7354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:39528 (ReqId={client: b9177b66c6b643048f5c550204e2c931, seq_no=5, attempt_no=9}) took 1002 ms. Trace:
I20250112 20:52:22.607296 7354 rpcz_store.cc:276] 0112 20:52:21.605024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:21.605111 (+ 87us) service_pool.cc:225] Handling call
0112 20:52:22.607133 (+1002022us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0112 20:52:21.605764 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd
0112 20:52:21.605887 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0112 20:52:21.605889 (+ 2us) write_op.cc:435] Acquired schema lock
0112 20:52:21.605891 (+ 2us) tablet.cc:659] Decoding operations
0112 20:52:21.605975 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0112 20:52:22.606834 (+1000859us) op_driver.cc:389] HandleFailure(Service unavailable: Write op should retry since it tries to acquire the partition lock that is held by another transaction that has higher txn ID)
0112 20:52:22.606854 (+ 20us) write_op.cc:489] Releasing partition, row and schema locks
0112 20:52:22.606864 (+ 10us) write_op.cc:454] Released schema lock
0112 20:52:22.606873 (+ 9us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":100}]]}
W20250112 20:52:22.618373 7320 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 3b59680f4f3c4b1e92ef2a30f3b3b2cd after 11 attempt(s): Failed to write to server: (no server available): Write(tablet: 3b59680f4f3c4b1e92ef2a30f3b3b2cd, num_ops: 1, num_attempts: 11) passed its deadline
I20250112 20:52:22.637624 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:22.647529 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:22.647816 416 tablet_replica.cc:331] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01: stopping tablet replica
I20250112 20:52:22.648041 416 raft_consensus.cc:2238] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:22.648247 416 raft_consensus.cc:2267] T 81c45183bcd848e6a249675d7961e0a1 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:22.649034 416 tablet_replica.cc:331] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01: stopping tablet replica
I20250112 20:52:22.649158 416 raft_consensus.cc:2238] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:22.649313 416 raft_consensus.cc:2267] T 3b59680f4f3c4b1e92ef2a30f3b3b2cd P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:22.649891 416 tablet_replica.cc:331] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01: stopping tablet replica
I20250112 20:52:22.650012 416 raft_consensus.cc:2238] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:22.650133 416 raft_consensus.cc:2267] T 3604da306304470285a571d6e4e9ae28 P 0282d3d0c17f42199a78da799f1a5c01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:22.654863 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:22.658965 416 master.cc:537] Master@127.0.104.62:40113 shutting down...
I20250112 20:52:22.668479 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:22.668773 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:22.668895 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2a77ee99523041d9aa8bb369eac61f96: stopping tablet replica
I20250112 20:52:22.682066 416 master.cc:559] Master@127.0.104.62:40113 shutdown complete.
[ OK ] TxnOpDispatcherITest.BeginTxnLockRetry (12048 ms)
[ RUN ] TxnOpDispatcherITest.ErrorInParticipantRegistration
I20250112 20:52:22.713434 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:40831
I20250112 20:52:22.713927 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:22.716238 7373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:22.716348 7374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:22.716348 7376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:22.716514 416 server_base.cc:1034] running on GCE node
I20250112 20:52:22.716959 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:22.717023 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:22.717056 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715142717049 us; error 0 us; skew 500 ppm
I20250112 20:52:22.717173 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:22.718339 416 webserver.cc:458] Webserver started at http://127.0.104.62:36609/ using document root <none> and password file <none>
I20250112 20:52:22.718545 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:22.718603 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:22.718696 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:22.719264 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "512fcf0864534638a5e3a2dc2616e139"
format_stamp: "Formatted at 2025-01-12 20:52:22 on dist-test-slave-3gql"
I20250112 20:52:22.721630 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:22.723140 7381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:22.723476 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:22.723567 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "512fcf0864534638a5e3a2dc2616e139"
format_stamp: "Formatted at 2025-01-12 20:52:22 on dist-test-slave-3gql"
I20250112 20:52:22.723655 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:22.731174 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:22.731734 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:22.744840 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40831
I20250112 20:52:22.744874 7442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40831 every 8 connection(s)
I20250112 20:52:22.746129 7443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:22.749915 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139: Bootstrap starting.
I20250112 20:52:22.751199 7443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:22.752774 7443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139: No bootstrap required, opened a new log
I20250112 20:52:22.753360 7443 raft_consensus.cc:357] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512fcf0864534638a5e3a2dc2616e139" member_type: VOTER }
I20250112 20:52:22.753484 7443 raft_consensus.cc:383] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:22.753545 7443 raft_consensus.cc:738] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512fcf0864534638a5e3a2dc2616e139, State: Initialized, Role: FOLLOWER
I20250112 20:52:22.753738 7443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512fcf0864534638a5e3a2dc2616e139" member_type: VOTER }
I20250112 20:52:22.753856 7443 raft_consensus.cc:397] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:22.753917 7443 raft_consensus.cc:491] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:22.753989 7443 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:22.755563 7443 raft_consensus.cc:513] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512fcf0864534638a5e3a2dc2616e139" member_type: VOTER }
I20250112 20:52:22.755731 7443 leader_election.cc:304] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 512fcf0864534638a5e3a2dc2616e139; no voters:
I20250112 20:52:22.756273 7443 leader_election.cc:290] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:22.756374 7446 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:22.756867 7446 raft_consensus.cc:695] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 1 LEADER]: Becoming Leader. State: Replica: 512fcf0864534638a5e3a2dc2616e139, State: Running, Role: LEADER
I20250112 20:52:22.757151 7443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:22.757125 7446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512fcf0864534638a5e3a2dc2616e139" member_type: VOTER }
I20250112 20:52:22.757997 7448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 512fcf0864534638a5e3a2dc2616e139. Latest consensus state: current_term: 1 leader_uuid: "512fcf0864534638a5e3a2dc2616e139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512fcf0864534638a5e3a2dc2616e139" member_type: VOTER } }
I20250112 20:52:22.758157 7448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:22.757964 7447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "512fcf0864534638a5e3a2dc2616e139" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "512fcf0864534638a5e3a2dc2616e139" member_type: VOTER } }
I20250112 20:52:22.758335 7447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:22.759137 7451 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:22.760243 7451 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:22.763278 7451 catalog_manager.cc:1348] Generated new cluster ID: 9b174adaccc048009d31edaf8f4d062f
I20250112 20:52:22.763352 7451 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:22.764585 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:22.773756 7451 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:22.774631 7451 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:22.786682 7451 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139: Generated new TSK 0
I20250112 20:52:22.786994 7451 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:22.796448 7398 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60700:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:22.797226 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:22.799751 7470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:22.800005 7471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:22.800315 7473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:22.800707 416 server_base.cc:1034] running on GCE node
I20250112 20:52:22.800992 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:22.801054 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:22.801091 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715142801084 us; error 0 us; skew 500 ppm
I20250112 20:52:22.801215 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:22.802479 416 webserver.cc:458] Webserver started at http://127.0.104.1:34999/ using document root <none> and password file <none>
I20250112 20:52:22.802690 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:22.802748 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:22.802853 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:22.803448 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "34d1662e31e84f3c9b1b8fd291964430"
format_stamp: "Formatted at 2025-01-12 20:52:22 on dist-test-slave-3gql"
I20250112 20:52:22.806270 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250112 20:52:22.807756 7478 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:22.808090 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:22.808187 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "34d1662e31e84f3c9b1b8fd291964430"
format_stamp: "Formatted at 2025-01-12 20:52:22 on dist-test-slave-3gql"
I20250112 20:52:22.808306 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:22.835477 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:22.836102 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:22.837917 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:22.838076 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:22.838184 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:22.838243 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:22.855741 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46807
I20250112 20:52:22.855764 7547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46807 every 8 connection(s)
I20250112 20:52:22.857260 7548 heartbeater.cc:346] Connected to a master server at 127.0.104.62:40831
I20250112 20:52:22.857373 7548 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:22.857681 7548 heartbeater.cc:510] Master 127.0.104.62:40831 requested a full tablet report, sending...
I20250112 20:52:22.858451 7398 ts_manager.cc:194] Registered new tserver with Master: 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1:46807)
I20250112 20:52:22.858947 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002271994s
I20250112 20:52:22.859424 7398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60714
I20250112 20:52:22.869650 7398 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60722:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:22.883016 7508 tablet_service.cc:1467] Processing CreateTablet for tablet 21ed8ac124fe485b8474d30a2cfb33be (DEFAULT_TABLE table=txn_op_dispatcher_test [id=b1e38467cd0d47b49246e458060ec599]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:22.883016 7507 tablet_service.cc:1467] Processing CreateTablet for tablet b2f6d1c63d434e7f858887800ab46ee9 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=b1e38467cd0d47b49246e458060ec599]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:22.883597 7508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21ed8ac124fe485b8474d30a2cfb33be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:22.883832 7507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2f6d1c63d434e7f858887800ab46ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:22.889982 7558 tablet_bootstrap.cc:492] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430: Bootstrap starting.
I20250112 20:52:22.891419 7558 tablet_bootstrap.cc:654] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:22.893045 7558 tablet_bootstrap.cc:492] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430: No bootstrap required, opened a new log
I20250112 20:52:22.893203 7558 ts_tablet_manager.cc:1397] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:22.893757 7558 raft_consensus.cc:357] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.893906 7558 raft_consensus.cc:383] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:22.893977 7558 raft_consensus.cc:738] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d1662e31e84f3c9b1b8fd291964430, State: Initialized, Role: FOLLOWER
I20250112 20:52:22.894181 7558 consensus_queue.cc:260] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.894351 7558 raft_consensus.cc:397] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:22.894423 7558 raft_consensus.cc:491] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:22.894502 7558 raft_consensus.cc:3054] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:22.896284 7558 raft_consensus.cc:513] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.896502 7558 leader_election.cc:304] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34d1662e31e84f3c9b1b8fd291964430; no voters:
I20250112 20:52:22.896965 7558 leader_election.cc:290] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:22.897126 7560 raft_consensus.cc:2798] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:22.897640 7548 heartbeater.cc:502] Master 127.0.104.62:40831 was elected leader, sending a full tablet report...
I20250112 20:52:22.897719 7558 ts_tablet_manager.cc:1428] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250112 20:52:22.897636 7560 raft_consensus.cc:695] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 1 LEADER]: Becoming Leader. State: Replica: 34d1662e31e84f3c9b1b8fd291964430, State: Running, Role: LEADER
I20250112 20:52:22.898011 7558 tablet_bootstrap.cc:492] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430: Bootstrap starting.
I20250112 20:52:22.898144 7560 consensus_queue.cc:237] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.899637 7558 tablet_bootstrap.cc:654] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:22.900507 7398 catalog_manager.cc:5526] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "34d1662e31e84f3c9b1b8fd291964430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:22.901535 7558 tablet_bootstrap.cc:492] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430: No bootstrap required, opened a new log
I20250112 20:52:22.901688 7558 ts_tablet_manager.cc:1397] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250112 20:52:22.902344 7558 raft_consensus.cc:357] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.902480 7558 raft_consensus.cc:383] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:22.902601 7558 raft_consensus.cc:738] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d1662e31e84f3c9b1b8fd291964430, State: Initialized, Role: FOLLOWER
I20250112 20:52:22.902832 7558 consensus_queue.cc:260] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.903013 7558 raft_consensus.cc:397] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:22.903136 7558 raft_consensus.cc:491] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:22.903247 7558 raft_consensus.cc:3054] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:22.905176 7558 raft_consensus.cc:513] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.905428 7558 leader_election.cc:304] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34d1662e31e84f3c9b1b8fd291964430; no voters:
I20250112 20:52:22.905681 7558 leader_election.cc:290] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:22.905750 7560 raft_consensus.cc:2798] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:22.906014 7560 raft_consensus.cc:695] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 LEADER]: Becoming Leader. State: Replica: 34d1662e31e84f3c9b1b8fd291964430, State: Running, Role: LEADER
I20250112 20:52:22.906167 7558 ts_tablet_manager.cc:1428] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250112 20:52:22.906252 7560 consensus_queue.cc:237] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:22.908033 7398 catalog_manager.cc:5526] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "34d1662e31e84f3c9b1b8fd291964430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:22.919871 7552 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:23.814983 7398 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60736:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:23.821648 7508 tablet_service.cc:1467] Processing CreateTablet for tablet 2e332f0fdccb415f981d55205602a4e5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=909f663853bc4a86a9708b819e14ffe1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:23.822124 7508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e332f0fdccb415f981d55205602a4e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:23.826277 7576 tablet_bootstrap.cc:492] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: Bootstrap starting.
I20250112 20:52:23.827754 7576 tablet_bootstrap.cc:654] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:23.829277 7576 tablet_bootstrap.cc:492] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: No bootstrap required, opened a new log
I20250112 20:52:23.829396 7576 ts_tablet_manager.cc:1397] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:23.829942 7576 raft_consensus.cc:357] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:23.830083 7576 raft_consensus.cc:383] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:23.830178 7576 raft_consensus.cc:738] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34d1662e31e84f3c9b1b8fd291964430, State: Initialized, Role: FOLLOWER
I20250112 20:52:23.830394 7576 consensus_queue.cc:260] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:23.830580 7576 raft_consensus.cc:397] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:23.830673 7576 raft_consensus.cc:491] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:23.830787 7576 raft_consensus.cc:3054] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:23.832322 7576 raft_consensus.cc:513] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:23.832532 7576 leader_election.cc:304] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34d1662e31e84f3c9b1b8fd291964430; no voters:
I20250112 20:52:23.832978 7576 leader_election.cc:290] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:23.833141 7578 raft_consensus.cc:2798] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:23.833561 7578 raft_consensus.cc:695] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 LEADER]: Becoming Leader. State: Replica: 34d1662e31e84f3c9b1b8fd291964430, State: Running, Role: LEADER
I20250112 20:52:23.833652 7576 ts_tablet_manager.cc:1428] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250112 20:52:23.833885 7578 consensus_queue.cc:237] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } }
I20250112 20:52:23.834777 7580 tablet_replica.cc:440] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: TxnStatusTablet state changed. Reason: New leader 34d1662e31e84f3c9b1b8fd291964430. Latest consensus state: current_term: 1 leader_uuid: "34d1662e31e84f3c9b1b8fd291964430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } } }
I20250112 20:52:23.834872 7579 tablet_replica.cc:440] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34d1662e31e84f3c9b1b8fd291964430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } } }
I20250112 20:52:23.835109 7579 tablet_replica.cc:443] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:23.835000 7580 tablet_replica.cc:443] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:23.836571 7583 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:23.836402 7397 catalog_manager.cc:5526] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "34d1662e31e84f3c9b1b8fd291964430" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34d1662e31e84f3c9b1b8fd291964430" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46807 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:23.836719 7583 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250112 20:52:24.086906 7589 tablet_replica.cc:1307] Invalid argument: Failed to write to server: 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1:46807): transaction ID 10 not found, current highest txn ID: 0: cancelling pending write operations [suppressed 1 similar messages]
W20250112 20:52:24.087060 7589 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33540: Invalid argument: Failed to write to server: 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1:46807): transaction ID 10 not found, current highest txn ID: 0
W20250112 20:52:24.087608 7552 batcher.cc:441] Invalid argument: Failed to write batch of 1 ops to tablet 21ed8ac124fe485b8474d30a2cfb33be after 1 attempt(s): Failed to write to server: 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1:46807): Failed to write to server: 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1:46807): transaction ID 10 not found, current highest txn ID: 0
W20250112 20:52:24.120654 7589 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33540: Illegal state: Failed to write to server: 34d1662e31e84f3c9b1b8fd291964430 (127.0.104.1:46807): transaction ID 1 not open: COMMITTED
I20250112 20:52:24.123229 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:24.131464 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:24.132021 416 tablet_replica.cc:331] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430: stopping tablet replica
I20250112 20:52:24.132269 416 raft_consensus.cc:2238] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:24.132503 416 raft_consensus.cc:2267] T 2e332f0fdccb415f981d55205602a4e5 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:24.133455 416 tablet_replica.cc:331] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430: stopping tablet replica
I20250112 20:52:24.133633 416 raft_consensus.cc:2238] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:24.133792 416 raft_consensus.cc:2267] T 21ed8ac124fe485b8474d30a2cfb33be P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:24.134264 416 tablet_replica.cc:331] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430: stopping tablet replica
I20250112 20:52:24.134395 416 raft_consensus.cc:2238] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:24.134545 416 raft_consensus.cc:2267] T b2f6d1c63d434e7f858887800ab46ee9 P 34d1662e31e84f3c9b1b8fd291964430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:24.149088 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:24.153461 416 master.cc:537] Master@127.0.104.62:40831 shutting down...
I20250112 20:52:24.161912 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:24.162137 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:24.162235 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 512fcf0864534638a5e3a2dc2616e139: stopping tablet replica
I20250112 20:52:24.166020 416 master.cc:559] Master@127.0.104.62:40831 shutdown complete.
[ OK ] TxnOpDispatcherITest.ErrorInParticipantRegistration (1483 ms)
[ RUN ] TxnOpDispatcherITest.ErrorInProcessingWriteOp
I20250112 20:52:24.197427 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:41835
I20250112 20:52:24.197947 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:24.200209 7598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:24.200353 7599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:24.200878 7601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:24.201010 416 server_base.cc:1034] running on GCE node
I20250112 20:52:24.201397 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:24.201457 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:24.201488 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715144201477 us; error 0 us; skew 500 ppm
I20250112 20:52:24.201615 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:24.202898 416 webserver.cc:458] Webserver started at http://127.0.104.62:40257/ using document root <none> and password file <none>
I20250112 20:52:24.203094 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:24.203150 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:24.203249 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:24.203810 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6"
format_stamp: "Formatted at 2025-01-12 20:52:24 on dist-test-slave-3gql"
I20250112 20:52:24.206302 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250112 20:52:24.207777 7606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:24.208127 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:24.208222 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6"
format_stamp: "Formatted at 2025-01-12 20:52:24 on dist-test-slave-3gql"
I20250112 20:52:24.208341 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:24.219358 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:24.219926 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:24.233879 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41835
I20250112 20:52:24.233891 7667 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41835 every 8 connection(s)
I20250112 20:52:24.235095 7668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:24.238844 7668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6: Bootstrap starting.
I20250112 20:52:24.240108 7668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:24.241618 7668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6: No bootstrap required, opened a new log
I20250112 20:52:24.242182 7668 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" member_type: VOTER }
I20250112 20:52:24.242306 7668 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:24.242342 7668 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2569dcc4bfa54da5b4e0f9be1f7decb6, State: Initialized, Role: FOLLOWER
I20250112 20:52:24.242544 7668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" member_type: VOTER }
I20250112 20:52:24.242660 7668 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:24.242722 7668 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:24.242800 7668 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:24.244421 7668 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" member_type: VOTER }
I20250112 20:52:24.244630 7668 leader_election.cc:304] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2569dcc4bfa54da5b4e0f9be1f7decb6; no voters:
I20250112 20:52:24.245031 7668 leader_election.cc:290] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:24.245152 7671 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:24.245568 7671 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 1 LEADER]: Becoming Leader. State: Replica: 2569dcc4bfa54da5b4e0f9be1f7decb6, State: Running, Role: LEADER
I20250112 20:52:24.245823 7671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" member_type: VOTER }
I20250112 20:52:24.245884 7668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:24.246659 7672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" member_type: VOTER } }
I20250112 20:52:24.246974 7672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:24.246724 7673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2569dcc4bfa54da5b4e0f9be1f7decb6. Latest consensus state: current_term: 1 leader_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2569dcc4bfa54da5b4e0f9be1f7decb6" member_type: VOTER } }
I20250112 20:52:24.247879 7673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:24.247915 7677 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:24.249254 7677 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:24.252071 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:24.252329 7677 catalog_manager.cc:1348] Generated new cluster ID: 05b2adee226a4b64a0987dac77a19a3b
I20250112 20:52:24.252404 7677 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:24.263586 7677 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:24.264272 7677 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:24.280216 7677 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6: Generated new TSK 0
I20250112 20:52:24.280479 7677 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:24.284618 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:24.287206 7695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:24.287400 7696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:24.287874 7698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:24.288632 416 server_base.cc:1034] running on GCE node
I20250112 20:52:24.288900 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:24.288975 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:24.289033 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715144289015 us; error 0 us; skew 500 ppm
I20250112 20:52:24.289211 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:24.290091 7623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:24.290691 416 webserver.cc:458] Webserver started at http://127.0.104.1:44721/ using document root <none> and password file <none>
I20250112 20:52:24.290944 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:24.291035 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:24.291188 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:24.291913 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "672adc3b9f9f4248878c05b4857fe68a"
format_stamp: "Formatted at 2025-01-12 20:52:24 on dist-test-slave-3gql"
I20250112 20:52:24.294608 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250112 20:52:24.296185 7703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:24.296538 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:24.296667 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "672adc3b9f9f4248878c05b4857fe68a"
format_stamp: "Formatted at 2025-01-12 20:52:24 on dist-test-slave-3gql"
I20250112 20:52:24.296772 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.ErrorInProcessingWriteOp.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:24.303575 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:24.304090 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:24.305719 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:24.305835 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:24.305971 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:24.306056 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:24.324469 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40587
I20250112 20:52:24.324519 7772 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40587 every 8 connection(s)
I20250112 20:52:24.326169 7773 heartbeater.cc:346] Connected to a master server at 127.0.104.62:41835
I20250112 20:52:24.326285 7773 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:24.326568 7773 heartbeater.cc:510] Master 127.0.104.62:41835 requested a full tablet report, sending...
I20250112 20:52:24.327260 7623 ts_manager.cc:194] Registered new tserver with Master: 672adc3b9f9f4248878c05b4857fe68a (127.0.104.1:40587)
I20250112 20:52:24.327818 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002271562s
I20250112 20:52:24.328302 7623 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36554
I20250112 20:52:24.337752 7623 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36570:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:24.349941 7733 tablet_service.cc:1467] Processing CreateTablet for tablet f02024223240488bbb61967f464d9776 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=7a840ec4f5b14f7289febb051044db48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:24.350425 7733 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f02024223240488bbb61967f464d9776. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:24.349967 7732 tablet_service.cc:1467] Processing CreateTablet for tablet 6925af19fa8449dca08edd29c7b9566a (DEFAULT_TABLE table=txn_op_dispatcher_test [id=7a840ec4f5b14f7289febb051044db48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:24.352952 7732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6925af19fa8449dca08edd29c7b9566a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:24.355162 7783 tablet_bootstrap.cc:492] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a: Bootstrap starting.
I20250112 20:52:24.356621 7783 tablet_bootstrap.cc:654] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:24.358393 7783 tablet_bootstrap.cc:492] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a: No bootstrap required, opened a new log
I20250112 20:52:24.358507 7783 ts_tablet_manager.cc:1397] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:52:24.359041 7783 raft_consensus.cc:357] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.359171 7783 raft_consensus.cc:383] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:24.359233 7783 raft_consensus.cc:738] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672adc3b9f9f4248878c05b4857fe68a, State: Initialized, Role: FOLLOWER
I20250112 20:52:24.359411 7783 consensus_queue.cc:260] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.359524 7783 raft_consensus.cc:397] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:24.359586 7783 raft_consensus.cc:491] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:24.359663 7783 raft_consensus.cc:3054] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:24.361200 7783 raft_consensus.cc:513] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.361388 7783 leader_election.cc:304] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672adc3b9f9f4248878c05b4857fe68a; no voters:
I20250112 20:52:24.361744 7783 leader_election.cc:290] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:24.361848 7785 raft_consensus.cc:2798] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:24.362594 7773 heartbeater.cc:502] Master 127.0.104.62:41835 was elected leader, sending a full tablet report...
I20250112 20:52:24.362591 7783 ts_tablet_manager.cc:1428] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:24.362928 7785 raft_consensus.cc:695] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 1 LEADER]: Becoming Leader. State: Replica: 672adc3b9f9f4248878c05b4857fe68a, State: Running, Role: LEADER
I20250112 20:52:24.363059 7783 tablet_bootstrap.cc:492] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a: Bootstrap starting.
I20250112 20:52:24.363200 7785 consensus_queue.cc:237] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.364763 7783 tablet_bootstrap.cc:654] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:24.365439 7623 catalog_manager.cc:5526] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a reported cstate change: term changed from 0 to 1, leader changed from <none> to 672adc3b9f9f4248878c05b4857fe68a (127.0.104.1). New cstate: current_term: 1 leader_uuid: "672adc3b9f9f4248878c05b4857fe68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:24.366504 7783 tablet_bootstrap.cc:492] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a: No bootstrap required, opened a new log
I20250112 20:52:24.366637 7783 ts_tablet_manager.cc:1397] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250112 20:52:24.367190 7783 raft_consensus.cc:357] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.367357 7783 raft_consensus.cc:383] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:24.367440 7783 raft_consensus.cc:738] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672adc3b9f9f4248878c05b4857fe68a, State: Initialized, Role: FOLLOWER
I20250112 20:52:24.367729 7783 consensus_queue.cc:260] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.367934 7783 raft_consensus.cc:397] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:24.368052 7783 raft_consensus.cc:491] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:24.368172 7783 raft_consensus.cc:3054] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:24.370245 7783 raft_consensus.cc:513] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.370460 7783 leader_election.cc:304] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672adc3b9f9f4248878c05b4857fe68a; no voters:
I20250112 20:52:24.370645 7783 leader_election.cc:290] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:24.370687 7785 raft_consensus.cc:2798] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:24.370870 7785 raft_consensus.cc:695] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 1 LEADER]: Becoming Leader. State: Replica: 672adc3b9f9f4248878c05b4857fe68a, State: Running, Role: LEADER
I20250112 20:52:24.371060 7783 ts_tablet_manager.cc:1428] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250112 20:52:24.371093 7785 consensus_queue.cc:237] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:24.372815 7623 catalog_manager.cc:5526] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a reported cstate change: term changed from 0 to 1, leader changed from <none> to 672adc3b9f9f4248878c05b4857fe68a (127.0.104.1). New cstate: current_term: 1 leader_uuid: "672adc3b9f9f4248878c05b4857fe68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:24.378640 7779 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:25.303015 7622 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39992:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:25.310030 7732 tablet_service.cc:1467] Processing CreateTablet for tablet bbea3b0cb45747f28864747b69c8ca9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2aee67f174804caba9714f804a165e80]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:25.310448 7732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbea3b0cb45747f28864747b69c8ca9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:25.314605 7801 tablet_bootstrap.cc:492] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: Bootstrap starting.
I20250112 20:52:25.315932 7801 tablet_bootstrap.cc:654] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:25.317508 7801 tablet_bootstrap.cc:492] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: No bootstrap required, opened a new log
I20250112 20:52:25.317641 7801 ts_tablet_manager.cc:1397] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:25.318205 7801 raft_consensus.cc:357] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:25.318361 7801 raft_consensus.cc:383] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:25.318455 7801 raft_consensus.cc:738] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 672adc3b9f9f4248878c05b4857fe68a, State: Initialized, Role: FOLLOWER
I20250112 20:52:25.318675 7801 consensus_queue.cc:260] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:25.318845 7801 raft_consensus.cc:397] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:25.318955 7801 raft_consensus.cc:491] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:25.319069 7801 raft_consensus.cc:3054] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:25.320686 7801 raft_consensus.cc:513] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:25.320921 7801 leader_election.cc:304] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 672adc3b9f9f4248878c05b4857fe68a; no voters:
I20250112 20:52:25.321313 7801 leader_election.cc:290] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:25.321507 7803 raft_consensus.cc:2798] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:25.321938 7801 ts_tablet_manager.cc:1428] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250112 20:52:25.322202 7803 raft_consensus.cc:695] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 1 LEADER]: Becoming Leader. State: Replica: 672adc3b9f9f4248878c05b4857fe68a, State: Running, Role: LEADER
I20250112 20:52:25.322472 7803 consensus_queue.cc:237] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } }
I20250112 20:52:25.323521 7805 tablet_replica.cc:440] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: TxnStatusTablet state changed. Reason: New leader 672adc3b9f9f4248878c05b4857fe68a. Latest consensus state: current_term: 1 leader_uuid: "672adc3b9f9f4248878c05b4857fe68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } } }
I20250112 20:52:25.323724 7805 tablet_replica.cc:443] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:25.324030 7804 tablet_replica.cc:440] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "672adc3b9f9f4248878c05b4857fe68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } } }
I20250112 20:52:25.324294 7804 tablet_replica.cc:443] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:25.325011 7622 catalog_manager.cc:5526] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a reported cstate change: term changed from 0 to 1, leader changed from <none> to 672adc3b9f9f4248878c05b4857fe68a (127.0.104.1). New cstate: current_term: 1 leader_uuid: "672adc3b9f9f4248878c05b4857fe68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "672adc3b9f9f4248878c05b4857fe68a" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40587 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:25.326787 7807 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:25.326961 7807 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250112 20:52:26.499701 7812 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:50998: Not implemented: transactions may only insert
W20250112 20:52:26.500269 7778 batcher.cc:441] Not implemented: Failed to write batch of 1 ops to tablet f02024223240488bbb61967f464d9776 after 1 attempt(s): Failed to write to server: 672adc3b9f9f4248878c05b4857fe68a (127.0.104.1:40587): transactions may only insert
I20250112 20:52:26.556481 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:26.566406 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:26.566896 416 tablet_replica.cc:331] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a: stopping tablet replica
I20250112 20:52:26.567113 416 raft_consensus.cc:2238] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:26.567361 416 raft_consensus.cc:2267] T bbea3b0cb45747f28864747b69c8ca9f P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:26.568220 416 tablet_replica.cc:331] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a: stopping tablet replica
I20250112 20:52:26.568394 416 raft_consensus.cc:2238] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:26.568586 416 raft_consensus.cc:2267] T f02024223240488bbb61967f464d9776 P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:26.569344 416 tablet_replica.cc:331] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a: stopping tablet replica
I20250112 20:52:26.569473 416 raft_consensus.cc:2238] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:26.569661 416 raft_consensus.cc:2267] T 6925af19fa8449dca08edd29c7b9566a P 672adc3b9f9f4248878c05b4857fe68a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:26.574857 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:26.580024 416 master.cc:537] Master@127.0.104.62:41835 shutting down...
I20250112 20:52:26.602579 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:26.602885 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:26.603013 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2569dcc4bfa54da5b4e0f9be1f7decb6: stopping tablet replica
I20250112 20:52:26.616703 416 master.cc:559] Master@127.0.104.62:41835 shutdown complete.
[ OK ] TxnOpDispatcherITest.ErrorInProcessingWriteOp (2431 ms)
[ RUN ] TxnOpDispatcherITest.NoPendingWriteOps
I20250112 20:52:26.629828 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:46177
I20250112 20:52:26.630468 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:26.632922 7828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:26.633059 7827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:26.633757 416 server_base.cc:1034] running on GCE node
W20250112 20:52:26.633803 7830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:26.634255 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:26.634341 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:26.634372 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715146634368 us; error 0 us; skew 500 ppm
I20250112 20:52:26.634536 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:26.635955 416 webserver.cc:458] Webserver started at http://127.0.104.62:36259/ using document root <none> and password file <none>
I20250112 20:52:26.636215 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:26.636283 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:26.636425 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:26.637187 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "766386d280eb47be9d7104e0762215ec"
format_stamp: "Formatted at 2025-01-12 20:52:26 on dist-test-slave-3gql"
I20250112 20:52:26.640177 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250112 20:52:26.642081 7835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:26.642472 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:26.642577 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "766386d280eb47be9d7104e0762215ec"
format_stamp: "Formatted at 2025-01-12 20:52:26 on dist-test-slave-3gql"
I20250112 20:52:26.642724 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:26.653695 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:26.654382 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:26.670333 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46177
I20250112 20:52:26.670354 7896 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46177 every 8 connection(s)
I20250112 20:52:26.671895 7897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:26.675701 7897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec: Bootstrap starting.
I20250112 20:52:26.677234 7897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:26.678926 7897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec: No bootstrap required, opened a new log
I20250112 20:52:26.679560 7897 raft_consensus.cc:357] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766386d280eb47be9d7104e0762215ec" member_type: VOTER }
I20250112 20:52:26.679690 7897 raft_consensus.cc:383] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:26.679749 7897 raft_consensus.cc:738] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766386d280eb47be9d7104e0762215ec, State: Initialized, Role: FOLLOWER
I20250112 20:52:26.679960 7897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766386d280eb47be9d7104e0762215ec" member_type: VOTER }
I20250112 20:52:26.680073 7897 raft_consensus.cc:397] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:26.680158 7897 raft_consensus.cc:491] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:26.680236 7897 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:26.681915 7897 raft_consensus.cc:513] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766386d280eb47be9d7104e0762215ec" member_type: VOTER }
I20250112 20:52:26.682101 7897 leader_election.cc:304] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 766386d280eb47be9d7104e0762215ec; no voters:
I20250112 20:52:26.682652 7897 leader_election.cc:290] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:26.682796 7900 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:26.683543 7900 raft_consensus.cc:695] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 1 LEADER]: Becoming Leader. State: Replica: 766386d280eb47be9d7104e0762215ec, State: Running, Role: LEADER
I20250112 20:52:26.683813 7897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:26.683828 7900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766386d280eb47be9d7104e0762215ec" member_type: VOTER }
I20250112 20:52:26.684753 7902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766386d280eb47be9d7104e0762215ec. Latest consensus state: current_term: 1 leader_uuid: "766386d280eb47be9d7104e0762215ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766386d280eb47be9d7104e0762215ec" member_type: VOTER } }
I20250112 20:52:26.684886 7902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:26.685298 7901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "766386d280eb47be9d7104e0762215ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766386d280eb47be9d7104e0762215ec" member_type: VOTER } }
I20250112 20:52:26.685454 7901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:26.698081 7905 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:26.699790 7905 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:26.701532 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:26.703785 7905 catalog_manager.cc:1348] Generated new cluster ID: b650e5e5a57d4982b474799731373172
I20250112 20:52:26.703886 7905 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:26.727286 7905 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:26.728229 7905 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:26.737689 7905 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec: Generated new TSK 0
I20250112 20:52:26.737922 7905 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:26.745105 7852 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42042:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:26.767439 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:26.770623 7926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:26.771275 7925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:26.772590 7928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:26.772665 416 server_base.cc:1034] running on GCE node
I20250112 20:52:26.773097 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:26.773173 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:26.773223 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715146773215 us; error 0 us; skew 500 ppm
I20250112 20:52:26.773391 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:26.774852 416 webserver.cc:458] Webserver started at http://127.0.104.1:34295/ using document root <none> and password file <none>
I20250112 20:52:26.775126 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:26.775206 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:26.775343 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:26.776149 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "c85519c47aa048cab5ad2d162f026283"
format_stamp: "Formatted at 2025-01-12 20:52:26 on dist-test-slave-3gql"
I20250112 20:52:26.779451 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250112 20:52:26.781482 7933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:26.781886 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:26.781996 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "c85519c47aa048cab5ad2d162f026283"
format_stamp: "Formatted at 2025-01-12 20:52:26 on dist-test-slave-3gql"
I20250112 20:52:26.782119 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NoPendingWriteOps.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:26.797721 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:26.798327 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:26.800806 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:26.800951 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:26.801095 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:26.801203 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:26.827864 8002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35481 every 8 connection(s)
I20250112 20:52:26.827858 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35481
I20250112 20:52:26.829803 8003 heartbeater.cc:346] Connected to a master server at 127.0.104.62:46177
I20250112 20:52:26.829923 8003 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:26.830307 8003 heartbeater.cc:510] Master 127.0.104.62:46177 requested a full tablet report, sending...
I20250112 20:52:26.831125 7852 ts_manager.cc:194] Registered new tserver with Master: c85519c47aa048cab5ad2d162f026283 (127.0.104.1:35481)
I20250112 20:52:26.831292 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002337773s
I20250112 20:52:26.832587 7852 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42050
I20250112 20:52:26.842805 7852 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42062:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:26.857954 7962 tablet_service.cc:1467] Processing CreateTablet for tablet 6d098c89c1ad40d5b908705f7fcf627f (DEFAULT_TABLE table=txn_op_dispatcher_test [id=8ce5486f528a48eb89247a5e9b186778]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:26.857954 7963 tablet_service.cc:1467] Processing CreateTablet for tablet a9ef52621b514e51bba52b2aff35a778 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=8ce5486f528a48eb89247a5e9b186778]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:26.858477 7962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d098c89c1ad40d5b908705f7fcf627f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:26.858721 7963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9ef52621b514e51bba52b2aff35a778. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:26.864393 8013 tablet_bootstrap.cc:492] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283: Bootstrap starting.
I20250112 20:52:26.866560 8013 tablet_bootstrap.cc:654] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:26.868485 8013 tablet_bootstrap.cc:492] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283: No bootstrap required, opened a new log
I20250112 20:52:26.868649 8013 ts_tablet_manager.cc:1397] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20250112 20:52:26.869304 8013 raft_consensus.cc:357] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.869446 8013 raft_consensus.cc:383] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:26.869489 8013 raft_consensus.cc:738] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85519c47aa048cab5ad2d162f026283, State: Initialized, Role: FOLLOWER
I20250112 20:52:26.869704 8013 consensus_queue.cc:260] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.869936 8013 raft_consensus.cc:397] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:26.870008 8013 raft_consensus.cc:491] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:26.870107 8013 raft_consensus.cc:3054] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:26.872092 8013 raft_consensus.cc:513] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.872285 8013 leader_election.cc:304] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85519c47aa048cab5ad2d162f026283; no voters:
I20250112 20:52:26.872841 8013 leader_election.cc:290] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:26.872992 8015 raft_consensus.cc:2798] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:26.873564 8003 heartbeater.cc:502] Master 127.0.104.62:46177 was elected leader, sending a full tablet report...
I20250112 20:52:26.874218 8013 ts_tablet_manager.cc:1428] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20250112 20:52:26.874557 8013 tablet_bootstrap.cc:492] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283: Bootstrap starting.
I20250112 20:52:26.877104 8013 tablet_bootstrap.cc:654] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:26.878666 8015 raft_consensus.cc:695] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 1 LEADER]: Becoming Leader. State: Replica: c85519c47aa048cab5ad2d162f026283, State: Running, Role: LEADER
I20250112 20:52:26.879033 8015 consensus_queue.cc:237] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.879513 8013 tablet_bootstrap.cc:492] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283: No bootstrap required, opened a new log
I20250112 20:52:26.879705 8013 ts_tablet_manager.cc:1397] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:52:26.880642 8013 raft_consensus.cc:357] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.880851 8013 raft_consensus.cc:383] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:26.880944 8013 raft_consensus.cc:738] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85519c47aa048cab5ad2d162f026283, State: Initialized, Role: FOLLOWER
I20250112 20:52:26.881218 8013 consensus_queue.cc:260] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.881413 8013 raft_consensus.cc:397] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:26.881532 8013 raft_consensus.cc:491] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:26.881654 8013 raft_consensus.cc:3054] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:26.881860 7852 catalog_manager.cc:5526] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 reported cstate change: term changed from 0 to 1, leader changed from <none> to c85519c47aa048cab5ad2d162f026283 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c85519c47aa048cab5ad2d162f026283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:26.884943 8013 raft_consensus.cc:513] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.885416 8013 leader_election.cc:304] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85519c47aa048cab5ad2d162f026283; no voters:
I20250112 20:52:26.885723 8013 leader_election.cc:290] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:26.885775 8017 raft_consensus.cc:2798] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:26.886411 8013 ts_tablet_manager.cc:1428] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250112 20:52:26.892235 8017 raft_consensus.cc:695] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 1 LEADER]: Becoming Leader. State: Replica: c85519c47aa048cab5ad2d162f026283, State: Running, Role: LEADER
I20250112 20:52:26.892583 8017 consensus_queue.cc:237] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:26.895385 7852 catalog_manager.cc:5526] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 reported cstate change: term changed from 0 to 1, leader changed from <none> to c85519c47aa048cab5ad2d162f026283 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c85519c47aa048cab5ad2d162f026283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:26.907866 8006 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:27.757452 7852 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42070:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:27.772794 7963 tablet_service.cc:1467] Processing CreateTablet for tablet 44d63d05d8654c54a5252355fecedca2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c974c67a86814b6aa059d06bea009c5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:27.773448 7963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44d63d05d8654c54a5252355fecedca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:27.778165 8032 tablet_bootstrap.cc:492] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: Bootstrap starting.
I20250112 20:52:27.779556 8032 tablet_bootstrap.cc:654] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:27.781114 8032 tablet_bootstrap.cc:492] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: No bootstrap required, opened a new log
I20250112 20:52:27.781244 8032 ts_tablet_manager.cc:1397] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:27.781826 8032 raft_consensus.cc:357] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:27.781986 8032 raft_consensus.cc:383] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:27.782049 8032 raft_consensus.cc:738] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85519c47aa048cab5ad2d162f026283, State: Initialized, Role: FOLLOWER
I20250112 20:52:27.782248 8032 consensus_queue.cc:260] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:27.782416 8032 raft_consensus.cc:397] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:27.782490 8032 raft_consensus.cc:491] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:27.782595 8032 raft_consensus.cc:3054] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:27.784897 8032 raft_consensus.cc:513] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:27.785070 8032 leader_election.cc:304] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85519c47aa048cab5ad2d162f026283; no voters:
I20250112 20:52:27.785408 8032 leader_election.cc:290] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:27.785527 8034 raft_consensus.cc:2798] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:27.786046 8032 ts_tablet_manager.cc:1428] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250112 20:52:27.786242 8034 raft_consensus.cc:695] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 1 LEADER]: Becoming Leader. State: Replica: c85519c47aa048cab5ad2d162f026283, State: Running, Role: LEADER
I20250112 20:52:27.786504 8034 consensus_queue.cc:237] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } }
I20250112 20:52:27.787439 8036 tablet_replica.cc:440] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: TxnStatusTablet state changed. Reason: New leader c85519c47aa048cab5ad2d162f026283. Latest consensus state: current_term: 1 leader_uuid: "c85519c47aa048cab5ad2d162f026283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } } }
I20250112 20:52:27.787633 8036 tablet_replica.cc:443] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:27.788512 8038 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:27.788699 8038 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:27.788919 7852 catalog_manager.cc:5526] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 reported cstate change: term changed from 0 to 1, leader changed from <none> to c85519c47aa048cab5ad2d162f026283 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c85519c47aa048cab5ad2d162f026283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:27.789831 8035 tablet_replica.cc:440] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c85519c47aa048cab5ad2d162f026283" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85519c47aa048cab5ad2d162f026283" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35481 } } }
I20250112 20:52:27.790009 8035 tablet_replica.cc:443] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:28.705912 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:28.715539 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:28.715901 416 tablet_replica.cc:331] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283: stopping tablet replica
I20250112 20:52:28.716174 416 raft_consensus.cc:2238] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:28.716388 416 raft_consensus.cc:2267] T 44d63d05d8654c54a5252355fecedca2 P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:28.717204 416 tablet_replica.cc:331] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283: stopping tablet replica
I20250112 20:52:28.717363 416 raft_consensus.cc:2238] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:28.717525 416 raft_consensus.cc:2267] T a9ef52621b514e51bba52b2aff35a778 P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:28.718187 416 tablet_replica.cc:331] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283: stopping tablet replica
I20250112 20:52:28.718325 416 raft_consensus.cc:2238] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:28.718461 416 raft_consensus.cc:2267] T 6d098c89c1ad40d5b908705f7fcf627f P c85519c47aa048cab5ad2d162f026283 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:28.723182 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:28.727221 416 master.cc:537] Master@127.0.104.62:46177 shutting down...
I20250112 20:52:28.736043 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:28.736251 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:28.736351 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 766386d280eb47be9d7104e0762215ec: stopping tablet replica
I20250112 20:52:28.749322 416 master.cc:559] Master@127.0.104.62:46177 shutdown complete.
[ OK ] TxnOpDispatcherITest.NoPendingWriteOps (2131 ms)
[ RUN ] TxnOpDispatcherITest.NonTransactionalWrites
I20250112 20:52:28.760427 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:40123
I20250112 20:52:28.760965 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:28.763115 8059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:28.763217 8060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:28.763649 8062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:28.764056 416 server_base.cc:1034] running on GCE node
I20250112 20:52:28.764323 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:28.764395 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:28.764461 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715148764439 us; error 0 us; skew 500 ppm
I20250112 20:52:28.764786 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:28.766110 416 webserver.cc:458] Webserver started at http://127.0.104.62:45259/ using document root <none> and password file <none>
I20250112 20:52:28.766382 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:28.766480 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:28.766645 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:28.767350 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "8e020b41067642c28e8e8994cbe9a010"
format_stamp: "Formatted at 2025-01-12 20:52:28 on dist-test-slave-3gql"
I20250112 20:52:28.770206 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250112 20:52:28.771870 8067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:28.772244 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250112 20:52:28.772362 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "8e020b41067642c28e8e8994cbe9a010"
format_stamp: "Formatted at 2025-01-12 20:52:28 on dist-test-slave-3gql"
I20250112 20:52:28.772502 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:28.778194 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:28.778683 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:28.792192 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40123
I20250112 20:52:28.792213 8128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40123 every 8 connection(s)
I20250112 20:52:28.793529 8129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:28.797268 8129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010: Bootstrap starting.
I20250112 20:52:28.798564 8129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:28.800004 8129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010: No bootstrap required, opened a new log
I20250112 20:52:28.800606 8129 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e020b41067642c28e8e8994cbe9a010" member_type: VOTER }
I20250112 20:52:28.800743 8129 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:28.800801 8129 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e020b41067642c28e8e8994cbe9a010, State: Initialized, Role: FOLLOWER
I20250112 20:52:28.800961 8129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e020b41067642c28e8e8994cbe9a010" member_type: VOTER }
I20250112 20:52:28.801069 8129 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:28.801131 8129 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:28.801194 8129 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:28.802737 8129 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e020b41067642c28e8e8994cbe9a010" member_type: VOTER }
I20250112 20:52:28.802908 8129 leader_election.cc:304] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e020b41067642c28e8e8994cbe9a010; no voters:
I20250112 20:52:28.803269 8129 leader_election.cc:290] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:28.803386 8132 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:28.803820 8132 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 1 LEADER]: Becoming Leader. State: Replica: 8e020b41067642c28e8e8994cbe9a010, State: Running, Role: LEADER
I20250112 20:52:28.804107 8129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:28.804077 8132 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e020b41067642c28e8e8994cbe9a010" member_type: VOTER }
I20250112 20:52:28.805032 8133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e020b41067642c28e8e8994cbe9a010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e020b41067642c28e8e8994cbe9a010" member_type: VOTER } }
I20250112 20:52:28.805120 8134 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e020b41067642c28e8e8994cbe9a010. Latest consensus state: current_term: 1 leader_uuid: "8e020b41067642c28e8e8994cbe9a010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e020b41067642c28e8e8994cbe9a010" member_type: VOTER } }
I20250112 20:52:28.805323 8133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:28.805330 8134 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:28.806545 8137 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:28.808326 8137 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:28.809226 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:28.811445 8137 catalog_manager.cc:1348] Generated new cluster ID: 5b64892bf60c432eb0c482497a5ee30f
I20250112 20:52:28.811532 8137 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:28.824157 8137 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:28.824893 8137 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:28.831964 8137 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010: Generated new TSK 0
I20250112 20:52:28.832197 8137 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:28.837983 8084 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48354:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:28.841645 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:28.844095 8156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:28.844511 8157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:28.844823 416 server_base.cc:1034] running on GCE node
W20250112 20:52:28.845067 8159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:28.845424 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:28.845496 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:28.845562 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715148845539 us; error 0 us; skew 500 ppm
I20250112 20:52:28.845743 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:28.846971 416 webserver.cc:458] Webserver started at http://127.0.104.1:38725/ using document root <none> and password file <none>
I20250112 20:52:28.847234 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:28.847321 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:28.847472 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:28.848098 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "74a9440998e145dfb715aedf58a53f88"
format_stamp: "Formatted at 2025-01-12 20:52:28 on dist-test-slave-3gql"
I20250112 20:52:28.850591 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250112 20:52:28.852066 8164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:28.852404 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:28.852526 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "74a9440998e145dfb715aedf58a53f88"
format_stamp: "Formatted at 2025-01-12 20:52:28 on dist-test-slave-3gql"
I20250112 20:52:28.852694 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.NonTransactionalWrites.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:28.887907 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:28.888450 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:28.890144 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:28.890287 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:28.890374 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:28.890426 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:28.911166 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37275
I20250112 20:52:28.911199 8233 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37275 every 8 connection(s)
I20250112 20:52:28.912640 8234 heartbeater.cc:346] Connected to a master server at 127.0.104.62:40123
I20250112 20:52:28.912747 8234 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:28.913131 8234 heartbeater.cc:510] Master 127.0.104.62:40123 requested a full tablet report, sending...
I20250112 20:52:28.913919 8084 ts_manager.cc:194] Registered new tserver with Master: 74a9440998e145dfb715aedf58a53f88 (127.0.104.1:37275)
I20250112 20:52:28.914309 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002262453s
I20250112 20:52:28.914927 8084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48360
I20250112 20:52:28.924147 8084 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48384:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:28.935500 8199 tablet_service.cc:1467] Processing CreateTablet for tablet 2471e0759b6e4ecbace1c01caab48c0e (DEFAULT_TABLE table=txn_op_dispatcher_test [id=2e4f88d9affb4b61babd76edc49b5314]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:28.935587 8198 tablet_service.cc:1467] Processing CreateTablet for tablet 7a0a7dfb885c416cae5a4ce11cc1dbc7 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=2e4f88d9affb4b61babd76edc49b5314]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:28.935940 8199 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2471e0759b6e4ecbace1c01caab48c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:28.936190 8198 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a0a7dfb885c416cae5a4ce11cc1dbc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:28.940954 8244 tablet_bootstrap.cc:492] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88: Bootstrap starting.
I20250112 20:52:28.942034 8244 tablet_bootstrap.cc:654] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:28.943382 8244 tablet_bootstrap.cc:492] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88: No bootstrap required, opened a new log
I20250112 20:52:28.943687 8244 ts_tablet_manager.cc:1397] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:28.944222 8244 raft_consensus.cc:357] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.944350 8244 raft_consensus.cc:383] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:28.944412 8244 raft_consensus.cc:738] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a9440998e145dfb715aedf58a53f88, State: Initialized, Role: FOLLOWER
I20250112 20:52:28.944617 8244 consensus_queue.cc:260] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.944736 8244 raft_consensus.cc:397] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:28.944794 8244 raft_consensus.cc:491] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:28.944864 8244 raft_consensus.cc:3054] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:28.946398 8244 raft_consensus.cc:513] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.946573 8244 leader_election.cc:304] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74a9440998e145dfb715aedf58a53f88; no voters:
I20250112 20:52:28.946941 8244 leader_election.cc:290] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:28.947049 8246 raft_consensus.cc:2798] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:28.947503 8246 raft_consensus.cc:695] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 1 LEADER]: Becoming Leader. State: Replica: 74a9440998e145dfb715aedf58a53f88, State: Running, Role: LEADER
I20250112 20:52:28.947575 8234 heartbeater.cc:502] Master 127.0.104.62:40123 was elected leader, sending a full tablet report...
I20250112 20:52:28.947654 8244 ts_tablet_manager.cc:1428] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:28.947863 8244 tablet_bootstrap.cc:492] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88: Bootstrap starting.
I20250112 20:52:28.947788 8246 consensus_queue.cc:237] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.949137 8244 tablet_bootstrap.cc:654] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:28.950068 8084 catalog_manager.cc:5526] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a9440998e145dfb715aedf58a53f88 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "74a9440998e145dfb715aedf58a53f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:28.950716 8244 tablet_bootstrap.cc:492] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88: No bootstrap required, opened a new log
I20250112 20:52:28.950837 8244 ts_tablet_manager.cc:1397] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:28.951335 8244 raft_consensus.cc:357] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.951443 8244 raft_consensus.cc:383] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:28.951507 8244 raft_consensus.cc:738] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74a9440998e145dfb715aedf58a53f88, State: Initialized, Role: FOLLOWER
I20250112 20:52:28.951689 8244 consensus_queue.cc:260] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.951844 8244 raft_consensus.cc:397] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:28.951900 8244 raft_consensus.cc:491] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:28.951943 8244 raft_consensus.cc:3054] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:28.954152 8244 raft_consensus.cc:513] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.954355 8244 leader_election.cc:304] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74a9440998e145dfb715aedf58a53f88; no voters:
I20250112 20:52:28.954636 8244 leader_election.cc:290] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:28.954651 8246 raft_consensus.cc:2798] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:28.954861 8246 raft_consensus.cc:695] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 1 LEADER]: Becoming Leader. State: Replica: 74a9440998e145dfb715aedf58a53f88, State: Running, Role: LEADER
I20250112 20:52:28.955042 8244 ts_tablet_manager.cc:1428] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250112 20:52:28.955070 8246 consensus_queue.cc:237] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } }
I20250112 20:52:28.956617 8083 catalog_manager.cc:5526] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 reported cstate change: term changed from 0 to 1, leader changed from <none> to 74a9440998e145dfb715aedf58a53f88 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "74a9440998e145dfb715aedf58a53f88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74a9440998e145dfb715aedf58a53f88" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37275 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:28.974790 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:28.983609 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:28.983963 416 tablet_replica.cc:331] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88: stopping tablet replica
I20250112 20:52:28.984205 416 raft_consensus.cc:2238] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:28.984390 416 raft_consensus.cc:2267] T 2471e0759b6e4ecbace1c01caab48c0e P 74a9440998e145dfb715aedf58a53f88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:28.985189 416 tablet_replica.cc:331] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88: stopping tablet replica
I20250112 20:52:28.985327 416 raft_consensus.cc:2238] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:28.985491 416 raft_consensus.cc:2267] T 7a0a7dfb885c416cae5a4ce11cc1dbc7 P 74a9440998e145dfb715aedf58a53f88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:28.990178 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:28.993971 416 master.cc:537] Master@127.0.104.62:40123 shutting down...
I20250112 20:52:29.840824 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:29.841133 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:29.841260 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 8e020b41067642c28e8e8994cbe9a010: stopping tablet replica
I20250112 20:52:29.854624 416 master.cc:559] Master@127.0.104.62:40123 shutdown complete.
[ OK ] TxnOpDispatcherITest.NonTransactionalWrites (1103 ms)
[ RUN ] TxnOpDispatcherITest.PreliminaryTasksTimeout
I20250112 20:52:29.863979 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:46523
I20250112 20:52:29.864480 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:29.866741 8255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:29.866806 8256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:29.867241 416 server_base.cc:1034] running on GCE node
W20250112 20:52:29.867220 8258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:29.867571 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:29.867620 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:29.867681 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715149867650 us; error 0 us; skew 500 ppm
I20250112 20:52:29.867825 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:29.869055 416 webserver.cc:458] Webserver started at http://127.0.104.62:41673/ using document root <none> and password file <none>
I20250112 20:52:29.869271 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:29.869365 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:29.869495 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:29.870110 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "d2c0014caa474c8ba00632372236aa29"
format_stamp: "Formatted at 2025-01-12 20:52:29 on dist-test-slave-3gql"
I20250112 20:52:29.872423 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:29.873953 8263 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:29.874328 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:29.874436 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "d2c0014caa474c8ba00632372236aa29"
format_stamp: "Formatted at 2025-01-12 20:52:29 on dist-test-slave-3gql"
I20250112 20:52:29.874564 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:29.888137 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:29.888749 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:29.901232 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46523
I20250112 20:52:29.901269 8324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46523 every 8 connection(s)
I20250112 20:52:29.902458 8325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:29.906217 8325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29: Bootstrap starting.
I20250112 20:52:29.907311 8325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:29.908772 8325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29: No bootstrap required, opened a new log
I20250112 20:52:29.909339 8325 raft_consensus.cc:357] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c0014caa474c8ba00632372236aa29" member_type: VOTER }
I20250112 20:52:29.909461 8325 raft_consensus.cc:383] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:29.909544 8325 raft_consensus.cc:738] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c0014caa474c8ba00632372236aa29, State: Initialized, Role: FOLLOWER
I20250112 20:52:29.909732 8325 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c0014caa474c8ba00632372236aa29" member_type: VOTER }
I20250112 20:52:29.909857 8325 raft_consensus.cc:397] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:29.909924 8325 raft_consensus.cc:491] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:29.910025 8325 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:29.911463 8325 raft_consensus.cc:513] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c0014caa474c8ba00632372236aa29" member_type: VOTER }
I20250112 20:52:29.911664 8325 leader_election.cc:304] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c0014caa474c8ba00632372236aa29; no voters:
I20250112 20:52:29.912019 8325 leader_election.cc:290] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:29.912217 8328 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:29.912698 8328 raft_consensus.cc:695] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 1 LEADER]: Becoming Leader. State: Replica: d2c0014caa474c8ba00632372236aa29, State: Running, Role: LEADER
I20250112 20:52:29.912938 8325 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:29.912983 8328 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c0014caa474c8ba00632372236aa29" member_type: VOTER }
I20250112 20:52:29.913918 8330 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c0014caa474c8ba00632372236aa29. Latest consensus state: current_term: 1 leader_uuid: "d2c0014caa474c8ba00632372236aa29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c0014caa474c8ba00632372236aa29" member_type: VOTER } }
I20250112 20:52:29.914098 8330 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:29.913868 8329 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2c0014caa474c8ba00632372236aa29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c0014caa474c8ba00632372236aa29" member_type: VOTER } }
I20250112 20:52:29.914462 8329 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:29.915357 8333 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:29.916710 8333 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:29.917519 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:29.919524 8333 catalog_manager.cc:1348] Generated new cluster ID: 858b004ead944e208e775dbdb53c6110
I20250112 20:52:29.919618 8333 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:29.933485 8333 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:29.934194 8333 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:29.950949 8333 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29: Generated new TSK 0
I20250112 20:52:29.951195 8333 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:29.963346 8280 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45126:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:29.982630 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:29.985636 8352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:29.985730 8353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:29.986402 8355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:29.987123 416 server_base.cc:1034] running on GCE node
I20250112 20:52:29.987342 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:29.987417 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:29.987459 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715149987446 us; error 0 us; skew 500 ppm
I20250112 20:52:29.987594 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:29.988936 416 webserver.cc:458] Webserver started at http://127.0.104.1:36441/ using document root <none> and password file <none>
I20250112 20:52:29.989202 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:29.989275 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:29.989437 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:29.990128 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "ab098d0cb66341f6a950866dc6a96f3d"
format_stamp: "Formatted at 2025-01-12 20:52:29 on dist-test-slave-3gql"
I20250112 20:52:29.992681 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250112 20:52:29.994354 8360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:29.994767 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250112 20:52:29.994892 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "ab098d0cb66341f6a950866dc6a96f3d"
format_stamp: "Formatted at 2025-01-12 20:52:29 on dist-test-slave-3gql"
I20250112 20:52:29.995004 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.PreliminaryTasksTimeout.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:30.009229 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:30.009728 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:30.011294 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:30.011466 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:30.011602 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:30.011680 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:30.030828 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:42363
I20250112 20:52:30.030840 8429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:42363 every 8 connection(s)
I20250112 20:52:30.032330 8430 heartbeater.cc:346] Connected to a master server at 127.0.104.62:46523
I20250112 20:52:30.032451 8430 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:30.032791 8430 heartbeater.cc:510] Master 127.0.104.62:46523 requested a full tablet report, sending...
I20250112 20:52:30.033522 8280 ts_manager.cc:194] Registered new tserver with Master: ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1:42363)
I20250112 20:52:30.034053 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002291636s
I20250112 20:52:30.034636 8280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45132
I20250112 20:52:30.047632 8280 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45154:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:30.060673 8389 tablet_service.cc:1467] Processing CreateTablet for tablet a076ad7ac6974dabae5cf53f9404486b (DEFAULT_TABLE table=txn_op_dispatcher_test [id=4e96786576504758b3d2963e5f8ba6c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:30.060719 8390 tablet_service.cc:1467] Processing CreateTablet for tablet 466eab9fc40a4d1ca1c6f58d0ea71667 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=4e96786576504758b3d2963e5f8ba6c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:30.061156 8389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a076ad7ac6974dabae5cf53f9404486b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:30.061412 8390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 466eab9fc40a4d1ca1c6f58d0ea71667. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:30.065886 8440 tablet_bootstrap.cc:492] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d: Bootstrap starting.
I20250112 20:52:30.067157 8440 tablet_bootstrap.cc:654] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:30.068816 8440 tablet_bootstrap.cc:492] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d: No bootstrap required, opened a new log
I20250112 20:52:30.068949 8440 ts_tablet_manager.cc:1397] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:52:30.069563 8440 raft_consensus.cc:357] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.069726 8440 raft_consensus.cc:383] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:30.069833 8440 raft_consensus.cc:738] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab098d0cb66341f6a950866dc6a96f3d, State: Initialized, Role: FOLLOWER
I20250112 20:52:30.070113 8440 consensus_queue.cc:260] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.070279 8440 raft_consensus.cc:397] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:30.070351 8440 raft_consensus.cc:491] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:30.070462 8440 raft_consensus.cc:3054] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:30.072028 8440 raft_consensus.cc:513] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.072283 8440 leader_election.cc:304] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab098d0cb66341f6a950866dc6a96f3d; no voters:
I20250112 20:52:30.072772 8440 leader_election.cc:290] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:30.073014 8442 raft_consensus.cc:2798] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:30.073532 8430 heartbeater.cc:502] Master 127.0.104.62:46523 was elected leader, sending a full tablet report...
I20250112 20:52:30.073537 8442 raft_consensus.cc:695] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 1 LEADER]: Becoming Leader. State: Replica: ab098d0cb66341f6a950866dc6a96f3d, State: Running, Role: LEADER
I20250112 20:52:30.073793 8442 consensus_queue.cc:237] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.074095 8440 ts_tablet_manager.cc:1428] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250112 20:52:30.074364 8440 tablet_bootstrap.cc:492] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d: Bootstrap starting.
I20250112 20:52:30.075999 8440 tablet_bootstrap.cc:654] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:30.076053 8280 catalog_manager.cc:5526] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ab098d0cb66341f6a950866dc6a96f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:30.077678 8440 tablet_bootstrap.cc:492] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d: No bootstrap required, opened a new log
I20250112 20:52:30.077836 8440 ts_tablet_manager.cc:1397] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:52:30.078569 8440 raft_consensus.cc:357] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.078781 8440 raft_consensus.cc:383] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:30.078891 8440 raft_consensus.cc:738] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab098d0cb66341f6a950866dc6a96f3d, State: Initialized, Role: FOLLOWER
I20250112 20:52:30.079150 8440 consensus_queue.cc:260] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.079339 8440 raft_consensus.cc:397] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:30.079442 8440 raft_consensus.cc:491] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:30.079541 8440 raft_consensus.cc:3054] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:30.081949 8440 raft_consensus.cc:513] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.082136 8440 leader_election.cc:304] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab098d0cb66341f6a950866dc6a96f3d; no voters:
I20250112 20:52:30.082365 8440 leader_election.cc:290] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:30.082412 8442 raft_consensus.cc:2798] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:30.082592 8442 raft_consensus.cc:695] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 1 LEADER]: Becoming Leader. State: Replica: ab098d0cb66341f6a950866dc6a96f3d, State: Running, Role: LEADER
I20250112 20:52:30.082847 8440 ts_tablet_manager.cc:1428] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:52:30.082851 8442 consensus_queue.cc:237] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.084620 8280 catalog_manager.cc:5526] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ab098d0cb66341f6a950866dc6a96f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:30.096705 8436 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:30.977865 8280 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45166:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:30.984809 8389 tablet_service.cc:1467] Processing CreateTablet for tablet f19fd23d071d429fbd81192f3d47fe18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f946d358f6974f07abb83345c5a478be]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:30.985272 8389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f19fd23d071d429fbd81192f3d47fe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:30.989473 8458 tablet_bootstrap.cc:492] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: Bootstrap starting.
I20250112 20:52:30.990646 8458 tablet_bootstrap.cc:654] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:30.992251 8458 tablet_bootstrap.cc:492] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: No bootstrap required, opened a new log
I20250112 20:52:30.992369 8458 ts_tablet_manager.cc:1397] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:30.992931 8458 raft_consensus.cc:357] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.993060 8458 raft_consensus.cc:383] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:30.993125 8458 raft_consensus.cc:738] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab098d0cb66341f6a950866dc6a96f3d, State: Initialized, Role: FOLLOWER
I20250112 20:52:30.993307 8458 consensus_queue.cc:260] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.993445 8458 raft_consensus.cc:397] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:30.993505 8458 raft_consensus.cc:491] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:30.993590 8458 raft_consensus.cc:3054] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:30.995088 8458 raft_consensus.cc:513] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.995262 8458 leader_election.cc:304] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab098d0cb66341f6a950866dc6a96f3d; no voters:
I20250112 20:52:30.995579 8458 leader_election.cc:290] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:30.995693 8460 raft_consensus.cc:2798] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:30.996186 8460 raft_consensus.cc:695] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 1 LEADER]: Becoming Leader. State: Replica: ab098d0cb66341f6a950866dc6a96f3d, State: Running, Role: LEADER
I20250112 20:52:30.996243 8458 ts_tablet_manager.cc:1428] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:30.996445 8460 consensus_queue.cc:237] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } }
I20250112 20:52:30.997560 8461 tablet_replica.cc:440] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab098d0cb66341f6a950866dc6a96f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } } }
I20250112 20:52:30.997936 8462 tablet_replica.cc:440] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: TxnStatusTablet state changed. Reason: New leader ab098d0cb66341f6a950866dc6a96f3d. Latest consensus state: current_term: 1 leader_uuid: "ab098d0cb66341f6a950866dc6a96f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } } }
I20250112 20:52:30.998001 8461 tablet_replica.cc:443] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:30.998121 8462 tablet_replica.cc:443] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:30.998541 8465 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:30.998718 8465 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:30.998950 8280 catalog_manager.cc:5526] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d reported cstate change: term changed from 0 to 1, leader changed from <none> to ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1). New cstate: current_term: 1 leader_uuid: "ab098d0cb66341f6a950866dc6a96f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab098d0cb66341f6a950866dc6a96f3d" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42363 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:32.041821 8469 meta_cache.cc:302] tablet 466eab9fc40a4d1ca1c6f58d0ea71667: replica ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1:42363) has failed: Timed out: Write RPC to 127.0.104.1:42363 timed out after 0.995s (SENT)
W20250112 20:52:32.042001 8469 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 466eab9fc40a4d1ca1c6f58d0ea71667 after 1 attempt(s): Failed to write to server: ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1:42363): Write RPC to 127.0.104.1:42363 timed out after 0.995s (SENT)
W20250112 20:52:33.049793 8483 tablet_replica.cc:1307] Timed out: time out prior registering tablet a076ad7ac6974dabae5cf53f9404486b as participant (txn ID 0): cancelling pending write operations [suppressed 1 similar messages]
W20250112 20:52:33.049973 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet a076ad7ac6974dabae5cf53f9404486b as participant (txn ID 0)
W20250112 20:52:33.050068 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet 466eab9fc40a4d1ca1c6f58d0ea71667 as participant (txn ID 0)
W20250112 20:52:33.050325 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=1, attempt_no=0}) took 2003 ms (client timeout 997 ms). Trace:
W20250112 20:52:33.050345 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=0, attempt_no=0}) took 2003 ms (client timeout 997 ms). Trace:
W20250112 20:52:33.050418 8484 rpcz_store.cc:269] 0112 20:52:31.046740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.046798 (+ 58us) service_pool.cc:225] Handling call
0112 20:52:33.050321 (+2003523us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":442,"threads_started":1,"txn-participant-registration.queue_time_us":523}
W20250112 20:52:33.050395 8483 rpcz_store.cc:269] 0112 20:52:31.046864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.046926 (+ 62us) service_pool.cc:225] Handling call
0112 20:52:33.050288 (+2003362us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":292,"threads_started":1,"txn-participant-registration.queue_time_us":408}
W20250112 20:52:33.050779 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet 466eab9fc40a4d1ca1c6f58d0ea71667 as participant (txn ID 0)
W20250112 20:52:33.050798 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet a076ad7ac6974dabae5cf53f9404486b as participant (txn ID 0)
W20250112 20:52:33.051017 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=2, attempt_no=0}) took 2004 ms (client timeout 996 ms). Trace:
W20250112 20:52:33.051062 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=3, attempt_no=0}) took 2004 ms (client timeout 996 ms). Trace:
W20250112 20:52:33.051123 8483 rpcz_store.cc:269] 0112 20:52:31.047061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.047481 (+ 420us) service_pool.cc:225] Handling call
0112 20:52:31.049898 (+ 2417us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x60d00052c330. stack: 00007fe4fcdf0fd6 00007fe4fcdf0d90 00007fe4fbbab48d 0000000000508a54 0000000000508a1d 00000000004e8e9a 00007fe5085115bc 00007fe508510ad0 00007fe50f1f70af 00007fe5092badc6 00007fe5092baaf5 00007fe50366530e 00007fe5036641ee 00007fe50366a425 00007fe50366ad4f 00007fe50366ab78
0112 20:52:33.051047 (+2001149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3841920}
W20250112 20:52:33.051112 8484 rpcz_store.cc:269] 0112 20:52:31.046962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.047032 (+ 70us) service_pool.cc:225] Handling call
0112 20:52:31.050069 (+ 3037us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x60d0004af470. stack: 00007fe4fcdf0fd6 00007fe4fcdf0d90 00007fe4fbbab48d 0000000000508a54 0000000000508a1d 00000000004e8e9a 00007fe5085115bc 00007fe508510ad0 00007fe50f1f70af 00007fe5092badc6 00007fe5092baaf5 00007fe50366530e 00007fe5036641ee 00007fe50366a425 00007fe50366ad4f 00007fe50366ab78
0112 20:52:33.051000 (+2000931us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6218496}
W20250112 20:52:33.051334 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet 466eab9fc40a4d1ca1c6f58d0ea71667 as participant (txn ID 0)
W20250112 20:52:33.051332 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet a076ad7ac6974dabae5cf53f9404486b as participant (txn ID 0)
W20250112 20:52:33.051573 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=5, attempt_no=0}) took 2004 ms (client timeout 996 ms). Trace:
W20250112 20:52:33.051600 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=4, attempt_no=0}) took 2004 ms (client timeout 996 ms). Trace:
W20250112 20:52:33.051656 8483 rpcz_store.cc:269] 0112 20:52:31.047146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.048031 (+ 885us) service_pool.cc:225] Handling call
0112 20:52:31.049919 (+ 1888us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x60d00052c330. stack: 00007fe4fcdf0fd6 00007fe4fcdf0d90 00007fe4fbbab48d 0000000000508a54 0000000000508a1d 00000000004e8e9a 00007fe5085115bc 00007fe508510ad0 00007fe50f1f70af 00007fe5092badc6 00007fe5092baaf5 00007fe50366530e 00007fe5036641ee 00007fe50366a425 00007fe50366ad4f 00007fe50366ab78
0112 20:52:33.051585 (+2001666us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2871552}
W20250112 20:52:33.051647 8484 rpcz_store.cc:269] 0112 20:52:31.047242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.047717 (+ 475us) service_pool.cc:225] Handling call
0112 20:52:31.050082 (+ 2365us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x60d0004af470. stack: 00007fe4fcdf0fd6 00007fe4fcdf0d90 00007fe4fbbab48d 0000000000508a54 0000000000508a1d 00000000004e8e9a 00007fe5085115bc 00007fe508510ad0 00007fe50f1f70af 00007fe5092badc6 00007fe5092baaf5 00007fe50366530e 00007fe5036641ee 00007fe50366a425 00007fe50366ad4f 00007fe50366ab78
0112 20:52:33.051557 (+2001475us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4806528}
W20250112 20:52:33.051851 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet a076ad7ac6974dabae5cf53f9404486b as participant (txn ID 0)
W20250112 20:52:33.051862 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32932: Timed out: time out prior registering tablet 466eab9fc40a4d1ca1c6f58d0ea71667 as participant (txn ID 0)
W20250112 20:52:33.052047 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=6, attempt_no=0}) took 2004 ms (client timeout 995 ms). Trace:
W20250112 20:52:33.052107 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32932 (ReqId={client: 9240fdc6c835483a984fb2aab1e2066c, seq_no=7, attempt_no=0}) took 2004 ms (client timeout 995 ms). Trace:
W20250112 20:52:33.052117 8483 rpcz_store.cc:269] 0112 20:52:31.047332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.047398 (+ 66us) service_pool.cc:225] Handling call
0112 20:52:31.049943 (+ 2545us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x60d00052c330. stack: 00007fe4fcdf0fd6 00007fe4fcdf0d90 00007fe4fbbab48d 0000000000508a54 0000000000508a1d 00000000004e8e9a 00007fe5085115bc 00007fe508510ad0 00007fe50f1f70af 00007fe5092badc6 00007fe5092baaf5 00007fe50366530e 00007fe5036641ee 00007fe50366a425 00007fe50366ad4f 00007fe50366ab78
0112 20:52:33.052034 (+2002091us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4635520}
W20250112 20:52:33.052212 8484 rpcz_store.cc:269] 0112 20:52:31.047432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:31.047588 (+ 156us) service_pool.cc:225] Handling call
0112 20:52:31.050114 (+ 2526us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x60d0004af470. stack: 00007fe4fcdf0fd6 00007fe4fcdf0d90 00007fe4fbbab48d 0000000000508a54 0000000000508a1d 00000000004e8e9a 00007fe5085115bc 00007fe508510ad0 00007fe50f1f70af 00007fe5092badc6 00007fe5092baaf5 00007fe50366530e 00007fe5036641ee 00007fe50366a425 00007fe50366ad4f 00007fe50366ab78
0112 20:52:33.052091 (+2001977us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4604544}
W20250112 20:52:34.110445 8493 meta_cache.cc:302] tablet 466eab9fc40a4d1ca1c6f58d0ea71667: replica ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1:42363) has failed: Timed out: Write RPC to 127.0.104.1:42363 timed out after 0.995s (SENT)
W20250112 20:52:34.110597 8493 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 466eab9fc40a4d1ca1c6f58d0ea71667 after 1 attempt(s): Failed to write to server: ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1:42363): Write RPC to 127.0.104.1:42363 timed out after 0.995s (SENT)
W20250112 20:52:35.127548 8483 tablet_replica.cc:1307] Timed out: time out prior submitting BEGIN_TXN for participant a076ad7ac6974dabae5cf53f9404486b (txn ID 1): cancelling pending write operations [suppressed 1 similar messages]
W20250112 20:52:35.127656 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant 466eab9fc40a4d1ca1c6f58d0ea71667 (txn ID 1)
W20250112 20:52:35.127705 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant a076ad7ac6974dabae5cf53f9404486b (txn ID 1)
W20250112 20:52:35.127961 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=0, attempt_no=0}) took 2011 ms (client timeout 998 ms). Trace:
W20250112 20:52:35.127941 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=3, attempt_no=0}) took 2011 ms (client timeout 996 ms). Trace:
W20250112 20:52:35.128070 8483 rpcz_store.cc:269] 0112 20:52:33.116797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.116900 (+ 103us) service_pool.cc:225] Handling call
0112 20:52:35.127921 (+2011021us) inbound_call.cc:173] Queueing success response
Metrics: {"txn-participant-registration.queue_time_us":49}
W20250112 20:52:35.128041 8484 rpcz_store.cc:269] 0112 20:52:33.116417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.116473 (+ 56us) service_pool.cc:225] Handling call
0112 20:52:35.127945 (+2011472us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87296,"txn-participant-registration.queue_time_us":63}
W20250112 20:52:35.128330 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant 466eab9fc40a4d1ca1c6f58d0ea71667 (txn ID 1)
W20250112 20:52:35.128345 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant a076ad7ac6974dabae5cf53f9404486b (txn ID 1)
W20250112 20:52:35.128481 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=1, attempt_no=0}) took 2011 ms (client timeout 998 ms). Trace:
W20250112 20:52:35.128509 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=6, attempt_no=0}) took 2011 ms (client timeout 995 ms). Trace:
W20250112 20:52:35.128568 8484 rpcz_store.cc:269] 0112 20:52:33.116507 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.116608 (+ 101us) service_pool.cc:225] Handling call
0112 20:52:35.128466 (+2011858us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 20:52:35.128605 8483 rpcz_store.cc:269] 0112 20:52:33.117098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.117173 (+ 75us) service_pool.cc:225] Handling call
0112 20:52:35.128495 (+2011322us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 20:52:35.128726 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant 466eab9fc40a4d1ca1c6f58d0ea71667 (txn ID 1)
W20250112 20:52:35.128732 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant a076ad7ac6974dabae5cf53f9404486b (txn ID 1)
W20250112 20:52:35.128881 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=2, attempt_no=0}) took 2012 ms (client timeout 996 ms). Trace:
W20250112 20:52:35.128907 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=4, attempt_no=0}) took 2012 ms (client timeout 996 ms). Trace:
W20250112 20:52:35.128966 8484 rpcz_store.cc:269] 0112 20:52:33.116657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.116730 (+ 73us) service_pool.cc:225] Handling call
0112 20:52:35.128868 (+2012138us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 20:52:35.128978 8483 rpcz_store.cc:269] 0112 20:52:33.116910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.117363 (+ 453us) service_pool.cc:225] Handling call
0112 20:52:35.128894 (+2011531us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 20:52:35.129110 8483 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant a076ad7ac6974dabae5cf53f9404486b (txn ID 1)
W20250112 20:52:35.129113 8484 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32942: Timed out: time out prior submitting BEGIN_TXN for participant 466eab9fc40a4d1ca1c6f58d0ea71667 (txn ID 1)
W20250112 20:52:35.129261 8483 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=7, attempt_no=0}) took 2012 ms (client timeout 994 ms). Trace:
W20250112 20:52:35.129267 8484 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:32942 (ReqId={client: 04c5f01321124397b389f9bc698fa0b6, seq_no=5, attempt_no=0}) took 2012 ms (client timeout 995 ms). Trace:
W20250112 20:52:35.129341 8484 rpcz_store.cc:269] 0112 20:52:33.117002 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.117057 (+ 55us) service_pool.cc:225] Handling call
0112 20:52:35.129258 (+2012201us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 20:52:35.129335 8483 rpcz_store.cc:269] 0112 20:52:33.117223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:33.117461 (+ 238us) service_pool.cc:225] Handling call
0112 20:52:35.129248 (+2011787us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250112 20:52:35.144932 8511 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32954: Illegal state: Transaction hasn't been successfully started
W20250112 20:52:35.145370 8511 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:32954: Illegal state: Transaction hasn't been successfully started
I20250112 20:52:35.145440 8394 txn_status_manager.cc:244] Participant a076ad7ac6974dabae5cf53f9404486b of txn 1 returned error for BEGIN_COMMIT op, aborting: Illegal state: Failed to write to server: ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1:42363): Transaction hasn't been successfully started
I20250112 20:52:35.145922 8394 txn_status_manager.cc:244] Participant 466eab9fc40a4d1ca1c6f58d0ea71667 of txn 1 returned error for BEGIN_COMMIT op, aborting: Illegal state: Failed to write to server: ab098d0cb66341f6a950866dc6a96f3d (127.0.104.1:42363): Transaction hasn't been successfully started
I20250112 20:52:35.146001 8394 txn_status_manager.cc:206] Scheduling write for ABORT_IN_PROGRESS for txn 1
I20250112 20:52:35.162086 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:35.173327 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:35.173928 416 tablet_replica.cc:331] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d: stopping tablet replica
I20250112 20:52:35.174119 416 raft_consensus.cc:2238] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:35.174310 416 raft_consensus.cc:2267] T f19fd23d071d429fbd81192f3d47fe18 P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:35.175006 416 tablet_replica.cc:331] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d: stopping tablet replica
I20250112 20:52:35.175132 416 raft_consensus.cc:2238] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:35.175269 416 raft_consensus.cc:2267] T a076ad7ac6974dabae5cf53f9404486b P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:35.175910 416 tablet_replica.cc:331] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d: stopping tablet replica
I20250112 20:52:35.176046 416 raft_consensus.cc:2238] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:35.176198 416 raft_consensus.cc:2267] T 466eab9fc40a4d1ca1c6f58d0ea71667 P ab098d0cb66341f6a950866dc6a96f3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:35.190595 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:35.194886 416 master.cc:537] Master@127.0.104.62:46523 shutting down...
I20250112 20:52:35.203217 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:35.203467 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:35.203583 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P d2c0014caa474c8ba00632372236aa29: stopping tablet replica
I20250112 20:52:35.222573 416 master.cc:559] Master@127.0.104.62:46523 shutdown complete.
[ OK ] TxnOpDispatcherITest.PreliminaryTasksTimeout (5368 ms)
[ RUN ] TxnOpDispatcherITest.DuplicateTxnParticipantRegistration
I20250112 20:52:35.232367 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:40795
I20250112 20:52:35.232882 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:35.234941 8519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:35.234970 8520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:35.235437 8522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:35.235716 416 server_base.cc:1034] running on GCE node
I20250112 20:52:35.235929 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:35.235977 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:35.235999 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715155235997 us; error 0 us; skew 500 ppm
I20250112 20:52:35.236115 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:35.241520 416 webserver.cc:458] Webserver started at http://127.0.104.62:38003/ using document root <none> and password file <none>
I20250112 20:52:35.241719 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:35.241771 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:35.241863 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:35.242470 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "e94996ced0c34c9fae96c501e9d5c47e"
format_stamp: "Formatted at 2025-01-12 20:52:35 on dist-test-slave-3gql"
I20250112 20:52:35.244792 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250112 20:52:35.246243 8527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:35.246553 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:35.246651 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "e94996ced0c34c9fae96c501e9d5c47e"
format_stamp: "Formatted at 2025-01-12 20:52:35 on dist-test-slave-3gql"
I20250112 20:52:35.246745 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:35.253366 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:35.253810 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:35.265684 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40795
I20250112 20:52:35.265720 8588 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40795 every 8 connection(s)
I20250112 20:52:35.267140 8589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:35.270901 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e: Bootstrap starting.
I20250112 20:52:35.271929 8589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:35.273320 8589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e: No bootstrap required, opened a new log
I20250112 20:52:35.273861 8589 raft_consensus.cc:357] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94996ced0c34c9fae96c501e9d5c47e" member_type: VOTER }
I20250112 20:52:35.273981 8589 raft_consensus.cc:383] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:35.274014 8589 raft_consensus.cc:738] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94996ced0c34c9fae96c501e9d5c47e, State: Initialized, Role: FOLLOWER
I20250112 20:52:35.274195 8589 consensus_queue.cc:260] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94996ced0c34c9fae96c501e9d5c47e" member_type: VOTER }
I20250112 20:52:35.274295 8589 raft_consensus.cc:397] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:35.274348 8589 raft_consensus.cc:491] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:35.274415 8589 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:35.275907 8589 raft_consensus.cc:513] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94996ced0c34c9fae96c501e9d5c47e" member_type: VOTER }
I20250112 20:52:35.276091 8589 leader_election.cc:304] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e94996ced0c34c9fae96c501e9d5c47e; no voters:
I20250112 20:52:35.276415 8589 leader_election.cc:290] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:35.276530 8592 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:35.276952 8592 raft_consensus.cc:695] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 1 LEADER]: Becoming Leader. State: Replica: e94996ced0c34c9fae96c501e9d5c47e, State: Running, Role: LEADER
I20250112 20:52:35.277197 8592 consensus_queue.cc:237] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94996ced0c34c9fae96c501e9d5c47e" member_type: VOTER }
I20250112 20:52:35.277252 8589 sys_catalog.cc:564] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:35.278033 8593 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e94996ced0c34c9fae96c501e9d5c47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94996ced0c34c9fae96c501e9d5c47e" member_type: VOTER } }
I20250112 20:52:35.278090 8594 sys_catalog.cc:455] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e94996ced0c34c9fae96c501e9d5c47e. Latest consensus state: current_term: 1 leader_uuid: "e94996ced0c34c9fae96c501e9d5c47e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e94996ced0c34c9fae96c501e9d5c47e" member_type: VOTER } }
I20250112 20:52:35.278295 8593 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:35.278369 8594 sys_catalog.cc:458] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:35.279237 8597 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:35.280246 8597 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:35.283210 8597 catalog_manager.cc:1348] Generated new cluster ID: 80122fd8748c4276a2bbaf16bb89e43d
I20250112 20:52:35.283296 8597 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:35.283357 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:35.315691 8597 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:35.316428 8597 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:35.328563 8597 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e: Generated new TSK 0
I20250112 20:52:35.328788 8597 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:35.335616 8544 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47524:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:35.348176 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:35.350770 8616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:35.350772 8617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:35.351037 8619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:35.351354 416 server_base.cc:1034] running on GCE node
I20250112 20:52:35.351554 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:35.351617 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:35.351651 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715155351647 us; error 0 us; skew 500 ppm
I20250112 20:52:35.351822 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:35.353058 416 webserver.cc:458] Webserver started at http://127.0.104.1:36819/ using document root <none> and password file <none>
I20250112 20:52:35.353269 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:35.353330 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:35.353448 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:35.354024 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "62ea8cb9f6ff4085a76792ee696bec39"
format_stamp: "Formatted at 2025-01-12 20:52:35 on dist-test-slave-3gql"
I20250112 20:52:35.356299 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:35.357797 8624 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:35.358114 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:35.358211 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "62ea8cb9f6ff4085a76792ee696bec39"
format_stamp: "Formatted at 2025-01-12 20:52:35 on dist-test-slave-3gql"
I20250112 20:52:35.358309 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.DuplicateTxnParticipantRegistration.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:35.368500 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:35.368971 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:35.370508 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:35.370582 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:35.370705 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:35.370766 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:35.389377 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:42329
I20250112 20:52:35.389384 8693 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:42329 every 8 connection(s)
I20250112 20:52:35.390929 8694 heartbeater.cc:346] Connected to a master server at 127.0.104.62:40795
I20250112 20:52:35.391049 8694 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:35.391372 8694 heartbeater.cc:510] Master 127.0.104.62:40795 requested a full tablet report, sending...
I20250112 20:52:35.392088 8544 ts_manager.cc:194] Registered new tserver with Master: 62ea8cb9f6ff4085a76792ee696bec39 (127.0.104.1:42329)
I20250112 20:52:35.392628 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002299025s
I20250112 20:52:35.393174 8544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47536
I20250112 20:52:35.402329 8544 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47546:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:35.414501 8654 tablet_service.cc:1467] Processing CreateTablet for tablet 2be0a3de46e04505a622da0ed75cbca4 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=1777213c03c144c28e0ea2cd8cf904e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:35.414525 8653 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6d27cfb84244039f4d251bd61d274f (DEFAULT_TABLE table=txn_op_dispatcher_test [id=1777213c03c144c28e0ea2cd8cf904e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:35.414981 8654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2be0a3de46e04505a622da0ed75cbca4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:35.415210 8653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6d27cfb84244039f4d251bd61d274f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:35.419508 8704 tablet_bootstrap.cc:492] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39: Bootstrap starting.
I20250112 20:52:35.420696 8704 tablet_bootstrap.cc:654] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:35.422004 8704 tablet_bootstrap.cc:492] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39: No bootstrap required, opened a new log
I20250112 20:52:35.422119 8704 ts_tablet_manager.cc:1397] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250112 20:52:35.422653 8704 raft_consensus.cc:357] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.422792 8704 raft_consensus.cc:383] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:35.422853 8704 raft_consensus.cc:738] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ea8cb9f6ff4085a76792ee696bec39, State: Initialized, Role: FOLLOWER
I20250112 20:52:35.423019 8704 consensus_queue.cc:260] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.423130 8704 raft_consensus.cc:397] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:35.423195 8704 raft_consensus.cc:491] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:35.423265 8704 raft_consensus.cc:3054] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:35.425025 8704 raft_consensus.cc:513] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.425212 8704 leader_election.cc:304] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ea8cb9f6ff4085a76792ee696bec39; no voters:
I20250112 20:52:35.425593 8704 leader_election.cc:290] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:35.425740 8706 raft_consensus.cc:2798] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:35.426187 8694 heartbeater.cc:502] Master 127.0.104.62:40795 was elected leader, sending a full tablet report...
I20250112 20:52:35.426214 8706 raft_consensus.cc:695] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 LEADER]: Becoming Leader. State: Replica: 62ea8cb9f6ff4085a76792ee696bec39, State: Running, Role: LEADER
I20250112 20:52:35.426213 8704 ts_tablet_manager.cc:1428] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20250112 20:52:35.426647 8704 tablet_bootstrap.cc:492] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39: Bootstrap starting.
I20250112 20:52:35.426687 8706 consensus_queue.cc:237] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.427868 8704 tablet_bootstrap.cc:654] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:35.429333 8704 tablet_bootstrap.cc:492] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39: No bootstrap required, opened a new log
I20250112 20:52:35.429452 8704 ts_tablet_manager.cc:1397] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250112 20:52:35.429910 8704 raft_consensus.cc:357] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.430006 8704 raft_consensus.cc:383] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:35.430037 8704 raft_consensus.cc:738] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ea8cb9f6ff4085a76792ee696bec39, State: Initialized, Role: FOLLOWER
I20250112 20:52:35.430181 8704 consensus_queue.cc:260] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.430303 8704 raft_consensus.cc:397] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:35.430338 8704 raft_consensus.cc:491] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:35.430380 8704 raft_consensus.cc:3054] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:35.430444 8544 catalog_manager.cc:5526] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62ea8cb9f6ff4085a76792ee696bec39 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "62ea8cb9f6ff4085a76792ee696bec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:35.432117 8704 raft_consensus.cc:513] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.432293 8704 leader_election.cc:304] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ea8cb9f6ff4085a76792ee696bec39; no voters:
I20250112 20:52:35.432473 8704 leader_election.cc:290] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:35.432535 8707 raft_consensus.cc:2798] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:35.432956 8704 ts_tablet_manager.cc:1428] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250112 20:52:35.435899 8707 raft_consensus.cc:695] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 LEADER]: Becoming Leader. State: Replica: 62ea8cb9f6ff4085a76792ee696bec39, State: Running, Role: LEADER
I20250112 20:52:35.436125 8707 consensus_queue.cc:237] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:35.438102 8544 catalog_manager.cc:5526] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62ea8cb9f6ff4085a76792ee696bec39 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "62ea8cb9f6ff4085a76792ee696bec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:35.443148 8697 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:36.347422 8543 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:36.354136 8653 tablet_service.cc:1467] Processing CreateTablet for tablet bbabacdc4cee4463b9d53c26a30bafaf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c15a5a61bcdb4ba0b27d318e5332d059]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:36.354570 8653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbabacdc4cee4463b9d53c26a30bafaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:36.359367 8723 tablet_bootstrap.cc:492] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: Bootstrap starting.
I20250112 20:52:36.360877 8723 tablet_bootstrap.cc:654] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:36.362351 8723 tablet_bootstrap.cc:492] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: No bootstrap required, opened a new log
I20250112 20:52:36.362478 8723 ts_tablet_manager.cc:1397] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:36.363029 8723 raft_consensus.cc:357] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:36.363160 8723 raft_consensus.cc:383] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:36.363229 8723 raft_consensus.cc:738] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ea8cb9f6ff4085a76792ee696bec39, State: Initialized, Role: FOLLOWER
I20250112 20:52:36.363435 8723 consensus_queue.cc:260] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:36.363557 8723 raft_consensus.cc:397] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:36.363608 8723 raft_consensus.cc:491] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:36.363680 8723 raft_consensus.cc:3054] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:36.365325 8723 raft_consensus.cc:513] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:36.365514 8723 leader_election.cc:304] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ea8cb9f6ff4085a76792ee696bec39; no voters:
I20250112 20:52:36.365996 8723 leader_election.cc:290] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:36.366117 8725 raft_consensus.cc:2798] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:36.366556 8725 raft_consensus.cc:695] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 LEADER]: Becoming Leader. State: Replica: 62ea8cb9f6ff4085a76792ee696bec39, State: Running, Role: LEADER
I20250112 20:52:36.366649 8723 ts_tablet_manager.cc:1428] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:36.366814 8725 consensus_queue.cc:237] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } }
I20250112 20:52:36.367816 8727 tablet_replica.cc:440] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: TxnStatusTablet state changed. Reason: New leader 62ea8cb9f6ff4085a76792ee696bec39. Latest consensus state: current_term: 1 leader_uuid: "62ea8cb9f6ff4085a76792ee696bec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } } }
I20250112 20:52:36.367985 8727 tablet_replica.cc:443] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:36.368938 8726 tablet_replica.cc:440] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62ea8cb9f6ff4085a76792ee696bec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } } }
I20250112 20:52:36.368993 8729 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:36.369076 8726 tablet_replica.cc:443] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:36.369151 8729 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:36.369151 8543 catalog_manager.cc:5526] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62ea8cb9f6ff4085a76792ee696bec39 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "62ea8cb9f6ff4085a76792ee696bec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ea8cb9f6ff4085a76792ee696bec39" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42329 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:37.437036 8736 meta_cache.cc:302] tablet 2be0a3de46e04505a622da0ed75cbca4: replica 62ea8cb9f6ff4085a76792ee696bec39 (127.0.104.1:42329) has failed: Timed out: Write RPC to 127.0.104.1:42329 timed out after 0.999s (SENT)
W20250112 20:52:37.437258 8736 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 2be0a3de46e04505a622da0ed75cbca4 after 1 attempt(s): Failed to write to server: 62ea8cb9f6ff4085a76792ee696bec39 (127.0.104.1:42329): Write RPC to 127.0.104.1:42329 timed out after 0.999s (SENT)
W20250112 20:52:38.451686 8740 tablet_replica.cc:1307] Timed out: time out prior submitting BEGIN_TXN for participant 2be0a3de46e04505a622da0ed75cbca4 (txn ID 0): cancelling pending write operations [suppressed 1 similar messages]
W20250112 20:52:38.451885 8740 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33410: Timed out: time out prior submitting BEGIN_TXN for participant 2be0a3de46e04505a622da0ed75cbca4 (txn ID 0)
W20250112 20:52:38.452212 8740 rpcz_store.cc:267] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33410 (ReqId={client: b255c9d22d4040709f5bd0c236287640, seq_no=0, attempt_no=0}) took 2011 ms (client timeout 998 ms). Trace:
W20250112 20:52:38.452304 8740 rpcz_store.cc:269] 0112 20:52:36.440965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:36.441027 (+ 62us) service_pool.cc:225] Handling call
0112 20:52:38.452160 (+2011133us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":190,"threads_started":1,"txn-participant-registration.queue_time_us":271}
I20250112 20:52:38.525074 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:38.538407 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:38.538793 416 tablet_replica.cc:331] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39: stopping tablet replica
I20250112 20:52:38.539037 416 raft_consensus.cc:2238] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:38.539261 416 raft_consensus.cc:2267] T bbabacdc4cee4463b9d53c26a30bafaf P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:38.540066 416 tablet_replica.cc:331] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39: stopping tablet replica
I20250112 20:52:38.540210 416 raft_consensus.cc:2238] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:38.540374 416 raft_consensus.cc:2267] T 9c6d27cfb84244039f4d251bd61d274f P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:38.541088 416 tablet_replica.cc:331] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39: stopping tablet replica
I20250112 20:52:38.541241 416 raft_consensus.cc:2238] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:38.541386 416 raft_consensus.cc:2267] T 2be0a3de46e04505a622da0ed75cbca4 P 62ea8cb9f6ff4085a76792ee696bec39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:38.555799 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:38.559962 416 master.cc:537] Master@127.0.104.62:40795 shutting down...
I20250112 20:52:38.567902 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:38.568174 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:38.568280 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P e94996ced0c34c9fae96c501e9d5c47e: stopping tablet replica
I20250112 20:52:38.581475 416 master.cc:559] Master@127.0.104.62:40795 shutdown complete.
[ OK ] TxnOpDispatcherITest.DuplicateTxnParticipantRegistration (3358 ms)
[ RUN ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered
I20250112 20:52:38.591727 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:46719
I20250112 20:52:38.592300 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:38.595110 8755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:38.597874 8756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:38.597949 416 server_base.cc:1034] running on GCE node
W20250112 20:52:38.598356 8758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:38.598664 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:38.598721 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:38.598748 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715158598745 us; error 0 us; skew 500 ppm
I20250112 20:52:38.598886 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:38.600067 416 webserver.cc:458] Webserver started at http://127.0.104.62:46873/ using document root <none> and password file <none>
I20250112 20:52:38.600272 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:38.600330 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:38.600442 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:38.601075 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "357e4ef959e848dda5a4362ce4e47860"
format_stamp: "Formatted at 2025-01-12 20:52:38 on dist-test-slave-3gql"
I20250112 20:52:38.603479 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250112 20:52:38.605137 8763 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:38.605480 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:38.605566 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "357e4ef959e848dda5a4362ce4e47860"
format_stamp: "Formatted at 2025-01-12 20:52:38 on dist-test-slave-3gql"
I20250112 20:52:38.605683 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:38.624236 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:38.624861 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:38.637517 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46719
I20250112 20:52:38.637568 8824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46719 every 8 connection(s)
I20250112 20:52:38.638826 8825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:38.642928 8825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860: Bootstrap starting.
I20250112 20:52:38.644255 8825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:38.645959 8825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860: No bootstrap required, opened a new log
I20250112 20:52:38.646638 8825 raft_consensus.cc:357] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e4ef959e848dda5a4362ce4e47860" member_type: VOTER }
I20250112 20:52:38.646764 8825 raft_consensus.cc:383] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:38.646834 8825 raft_consensus.cc:738] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357e4ef959e848dda5a4362ce4e47860, State: Initialized, Role: FOLLOWER
I20250112 20:52:38.647044 8825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e4ef959e848dda5a4362ce4e47860" member_type: VOTER }
I20250112 20:52:38.647167 8825 raft_consensus.cc:397] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:38.647229 8825 raft_consensus.cc:491] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:38.647305 8825 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:38.648983 8825 raft_consensus.cc:513] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e4ef959e848dda5a4362ce4e47860" member_type: VOTER }
I20250112 20:52:38.649163 8825 leader_election.cc:304] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 357e4ef959e848dda5a4362ce4e47860; no voters:
I20250112 20:52:38.649580 8825 leader_election.cc:290] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:38.649708 8828 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:38.650136 8828 raft_consensus.cc:695] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 1 LEADER]: Becoming Leader. State: Replica: 357e4ef959e848dda5a4362ce4e47860, State: Running, Role: LEADER
I20250112 20:52:38.650368 8828 consensus_queue.cc:237] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e4ef959e848dda5a4362ce4e47860" member_type: VOTER }
I20250112 20:52:38.650424 8825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:38.651237 8829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "357e4ef959e848dda5a4362ce4e47860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e4ef959e848dda5a4362ce4e47860" member_type: VOTER } }
I20250112 20:52:38.651299 8830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 357e4ef959e848dda5a4362ce4e47860. Latest consensus state: current_term: 1 leader_uuid: "357e4ef959e848dda5a4362ce4e47860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357e4ef959e848dda5a4362ce4e47860" member_type: VOTER } }
I20250112 20:52:38.651510 8829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:38.651602 8830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:38.656376 8833 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:38.656787 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:38.657711 8833 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:38.661216 8833 catalog_manager.cc:1348] Generated new cluster ID: 976fbb26d8514295b54a38c480901947
I20250112 20:52:38.661365 8833 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:38.675410 8833 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:38.676371 8833 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:38.684243 8833 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860: Generated new TSK 0
I20250112 20:52:38.684480 8833 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:38.689924 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:38.692876 8853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:38.693069 8852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:38.694613 416 server_base.cc:1034] running on GCE node
W20250112 20:52:38.694875 8855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:38.695240 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:38.695313 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:38.695353 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715158695349 us; error 0 us; skew 500 ppm
I20250112 20:52:38.695526 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:38.695528 8780 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49952:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:38.697067 416 webserver.cc:458] Webserver started at http://127.0.104.1:44159/ using document root <none> and password file <none>
I20250112 20:52:38.697279 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:38.697347 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:38.697436 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:38.698002 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2"
format_stamp: "Formatted at 2025-01-12 20:52:38 on dist-test-slave-3gql"
I20250112 20:52:38.700402 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:38.701997 8860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:38.702486 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:38.702593 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2"
format_stamp: "Formatted at 2025-01-12 20:52:38 on dist-test-slave-3gql"
I20250112 20:52:38.702697 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:38.719703 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:38.720283 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:38.721922 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:38.722038 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:38.722173 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:38.722249 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:38.737986 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46753
I20250112 20:52:38.738006 8929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46753 every 8 connection(s)
I20250112 20:52:38.740060 8930 heartbeater.cc:346] Connected to a master server at 127.0.104.62:46719
I20250112 20:52:38.740195 8930 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:38.740571 8930 heartbeater.cc:510] Master 127.0.104.62:46719 requested a full tablet report, sending...
I20250112 20:52:38.741526 8780 ts_manager.cc:194] Registered new tserver with Master: e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1:46753)
I20250112 20:52:38.741928 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.001180781s
I20250112 20:52:38.742580 8780 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49968
I20250112 20:52:38.752565 8780 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49992:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:38.766808 8889 tablet_service.cc:1467] Processing CreateTablet for tablet 5923939d703c4cb7be1517efed7aa205 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=5b18c923a67d4a83ba4e3363e3d52833]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:38.766808 8890 tablet_service.cc:1467] Processing CreateTablet for tablet ce0c027710ff4171b5a5ca800ab0b101 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=5b18c923a67d4a83ba4e3363e3d52833]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:38.767513 8889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5923939d703c4cb7be1517efed7aa205. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:38.767725 8890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce0c027710ff4171b5a5ca800ab0b101. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:38.773605 8940 tablet_bootstrap.cc:492] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2: Bootstrap starting.
I20250112 20:52:38.774852 8940 tablet_bootstrap.cc:654] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:38.776309 8940 tablet_bootstrap.cc:492] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2: No bootstrap required, opened a new log
I20250112 20:52:38.776409 8940 ts_tablet_manager.cc:1397] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:38.777011 8940 raft_consensus.cc:357] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.777144 8940 raft_consensus.cc:383] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:38.777189 8940 raft_consensus.cc:738] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114d980a9aa4e3ba05a4068dbc0d0e2, State: Initialized, Role: FOLLOWER
I20250112 20:52:38.777324 8940 consensus_queue.cc:260] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.777417 8940 raft_consensus.cc:397] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:38.777454 8940 raft_consensus.cc:491] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:38.777499 8940 raft_consensus.cc:3054] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:38.779145 8940 raft_consensus.cc:513] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.779312 8940 leader_election.cc:304] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e114d980a9aa4e3ba05a4068dbc0d0e2; no voters:
I20250112 20:52:38.779681 8940 leader_election.cc:290] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:38.779834 8942 raft_consensus.cc:2798] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:38.780298 8930 heartbeater.cc:502] Master 127.0.104.62:46719 was elected leader, sending a full tablet report...
I20250112 20:52:38.780293 8942 raft_consensus.cc:695] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 LEADER]: Becoming Leader. State: Replica: e114d980a9aa4e3ba05a4068dbc0d0e2, State: Running, Role: LEADER
I20250112 20:52:38.780330 8940 ts_tablet_manager.cc:1428] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250112 20:52:38.780642 8940 tablet_bootstrap.cc:492] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2: Bootstrap starting.
I20250112 20:52:38.780571 8942 consensus_queue.cc:237] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.781709 8940 tablet_bootstrap.cc:654] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:38.782704 8780 catalog_manager.cc:5526] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:38.783192 8940 tablet_bootstrap.cc:492] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2: No bootstrap required, opened a new log
I20250112 20:52:38.783290 8940 ts_tablet_manager.cc:1397] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:38.783998 8940 raft_consensus.cc:357] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.784121 8940 raft_consensus.cc:383] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:38.784210 8940 raft_consensus.cc:738] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114d980a9aa4e3ba05a4068dbc0d0e2, State: Initialized, Role: FOLLOWER
I20250112 20:52:38.784416 8940 consensus_queue.cc:260] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.784623 8940 raft_consensus.cc:397] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:38.784700 8940 raft_consensus.cc:491] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:38.784772 8940 raft_consensus.cc:3054] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:38.786685 8940 raft_consensus.cc:513] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.786882 8940 leader_election.cc:304] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e114d980a9aa4e3ba05a4068dbc0d0e2; no voters:
I20250112 20:52:38.787072 8940 leader_election.cc:290] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:38.787133 8942 raft_consensus.cc:2798] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:38.787341 8942 raft_consensus.cc:695] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 LEADER]: Becoming Leader. State: Replica: e114d980a9aa4e3ba05a4068dbc0d0e2, State: Running, Role: LEADER
I20250112 20:52:38.787482 8940 ts_tablet_manager.cc:1428] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20250112 20:52:38.787935 8942 consensus_queue.cc:237] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:38.789786 8780 catalog_manager.cc:5526] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:38.803141 8936 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:39.710759 8780 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50002:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:39.725035 8889 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee024eb93614c6390c91a46eff37789 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9700876d37b649a79e0a60f04b475fba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:39.725490 8889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ee024eb93614c6390c91a46eff37789. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:39.729780 8958 tablet_bootstrap.cc:492] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: Bootstrap starting.
I20250112 20:52:39.731307 8958 tablet_bootstrap.cc:654] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:39.732847 8958 tablet_bootstrap.cc:492] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: No bootstrap required, opened a new log
I20250112 20:52:39.732982 8958 ts_tablet_manager.cc:1397] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:52:39.733547 8958 raft_consensus.cc:357] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:39.733682 8958 raft_consensus.cc:383] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:39.733742 8958 raft_consensus.cc:738] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e114d980a9aa4e3ba05a4068dbc0d0e2, State: Initialized, Role: FOLLOWER
I20250112 20:52:39.733963 8958 consensus_queue.cc:260] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:39.734126 8958 raft_consensus.cc:397] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:39.734195 8958 raft_consensus.cc:491] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:39.734267 8958 raft_consensus.cc:3054] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:39.735824 8958 raft_consensus.cc:513] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:39.736003 8958 leader_election.cc:304] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e114d980a9aa4e3ba05a4068dbc0d0e2; no voters:
I20250112 20:52:39.736362 8958 leader_election.cc:290] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:39.736498 8960 raft_consensus.cc:2798] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:39.736997 8958 ts_tablet_manager.cc:1428] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:39.737285 8960 raft_consensus.cc:695] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 LEADER]: Becoming Leader. State: Replica: e114d980a9aa4e3ba05a4068dbc0d0e2, State: Running, Role: LEADER
I20250112 20:52:39.737531 8960 consensus_queue.cc:237] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } }
I20250112 20:52:39.738700 8961 tablet_replica.cc:440] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } } }
I20250112 20:52:39.739082 8961 tablet_replica.cc:443] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:39.739864 8965 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:39.739996 8965 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:39.740054 8780 catalog_manager.cc:5526] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:39.739338 8962 tablet_replica.cc:440] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: TxnStatusTablet state changed. Reason: New leader e114d980a9aa4e3ba05a4068dbc0d0e2. Latest consensus state: current_term: 1 leader_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e114d980a9aa4e3ba05a4068dbc0d0e2" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46753 } } }
I20250112 20:52:39.740674 8962 tablet_replica.cc:443] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: This TxnStatusTablet replica's current role is: LEADER
W20250112 20:52:42.363847 8975 tablet_replica.cc:1307] Illegal state: Failed to write to server: e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1:46753): transaction ID 0 not open: ABORTED: cancelling pending write operations
W20250112 20:52:42.363962 8975 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33364: Illegal state: Failed to write to server: e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1:46753): transaction ID 0 not open: ABORTED
I20250112 20:52:42.364120 8975 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33364 (ReqId={client: 23817302acb64212a88dafee1fe8e457, seq_no=0, attempt_no=0}) took 2009 ms. Trace:
I20250112 20:52:42.364183 8975 rpcz_store.cc:276] 0112 20:52:40.354553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:40.354612 (+ 59us) service_pool.cc:225] Handling call
0112 20:52:42.364094 (+2009482us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":210,"threads_started":1,"txn-participant-registration.queue_time_us":310}
W20250112 20:52:42.364653 8935 batcher.cc:441] Illegal state: Failed to write batch of 1 ops to tablet ce0c027710ff4171b5a5ca800ab0b101 after 1 attempt(s): Failed to write to server: e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1:46753): Failed to write to server: e114d980a9aa4e3ba05a4068dbc0d0e2 (127.0.104.1:46753): transaction ID 0 not open: ABORTED
I20250112 20:52:42.372617 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:42.382735 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:42.383044 416 tablet_replica.cc:331] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2: stopping tablet replica
I20250112 20:52:42.383204 416 raft_consensus.cc:2238] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:42.383409 416 raft_consensus.cc:2267] T 5ee024eb93614c6390c91a46eff37789 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:42.383947 416 tablet_replica.cc:331] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2: stopping tablet replica
I20250112 20:52:42.384088 416 raft_consensus.cc:2238] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:42.384208 416 raft_consensus.cc:2267] T 5923939d703c4cb7be1517efed7aa205 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:42.384630 416 tablet_replica.cc:331] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2: stopping tablet replica
I20250112 20:52:42.384740 416 raft_consensus.cc:2238] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:42.384858 416 raft_consensus.cc:2267] T ce0c027710ff4171b5a5ca800ab0b101 P e114d980a9aa4e3ba05a4068dbc0d0e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:42.398397 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:42.402858 416 master.cc:537] Master@127.0.104.62:46719 shutting down...
I20250112 20:52:42.411715 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:42.411948 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:42.412045 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 357e4ef959e848dda5a4362ce4e47860: stopping tablet replica
I20250112 20:52:42.425280 416 master.cc:559] Master@127.0.104.62:46719 shutdown complete.
[ OK ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantNotYetRegistered (3843 ms)
[ RUN ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered
I20250112 20:52:42.435134 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:37885
I20250112 20:52:42.435588 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:42.437695 8984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:42.437809 8985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:42.438150 8987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:42.438256 416 server_base.cc:1034] running on GCE node
I20250112 20:52:42.438553 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:42.438603 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:42.438637 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715162438633 us; error 0 us; skew 500 ppm
I20250112 20:52:42.438769 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:42.439901 416 webserver.cc:458] Webserver started at http://127.0.104.62:38261/ using document root <none> and password file <none>
I20250112 20:52:42.440124 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:42.440186 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:42.440306 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:42.440949 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "dee554c435fc4dc99d07d138915e7e8e"
format_stamp: "Formatted at 2025-01-12 20:52:42 on dist-test-slave-3gql"
I20250112 20:52:42.443323 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250112 20:52:42.444736 8992 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:42.445092 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:42.445175 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "dee554c435fc4dc99d07d138915e7e8e"
format_stamp: "Formatted at 2025-01-12 20:52:42 on dist-test-slave-3gql"
I20250112 20:52:42.445266 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:42.457389 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:42.457859 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:42.469290 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37885
I20250112 20:52:42.469318 9053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37885 every 8 connection(s)
I20250112 20:52:42.470486 9054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:42.474090 9054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e: Bootstrap starting.
I20250112 20:52:42.475111 9054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:42.476388 9054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e: No bootstrap required, opened a new log
I20250112 20:52:42.476969 9054 raft_consensus.cc:357] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee554c435fc4dc99d07d138915e7e8e" member_type: VOTER }
I20250112 20:52:42.477094 9054 raft_consensus.cc:383] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:42.477146 9054 raft_consensus.cc:738] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee554c435fc4dc99d07d138915e7e8e, State: Initialized, Role: FOLLOWER
I20250112 20:52:42.477320 9054 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee554c435fc4dc99d07d138915e7e8e" member_type: VOTER }
I20250112 20:52:42.477421 9054 raft_consensus.cc:397] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:42.477475 9054 raft_consensus.cc:491] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:42.477560 9054 raft_consensus.cc:3054] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:42.479000 9054 raft_consensus.cc:513] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee554c435fc4dc99d07d138915e7e8e" member_type: VOTER }
I20250112 20:52:42.479153 9054 leader_election.cc:304] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee554c435fc4dc99d07d138915e7e8e; no voters:
I20250112 20:52:42.479483 9054 leader_election.cc:290] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:42.479588 9057 raft_consensus.cc:2798] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:42.479980 9057 raft_consensus.cc:695] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 1 LEADER]: Becoming Leader. State: Replica: dee554c435fc4dc99d07d138915e7e8e, State: Running, Role: LEADER
I20250112 20:52:42.480262 9054 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:42.480213 9057 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee554c435fc4dc99d07d138915e7e8e" member_type: VOTER }
I20250112 20:52:42.481079 9058 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee554c435fc4dc99d07d138915e7e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee554c435fc4dc99d07d138915e7e8e" member_type: VOTER } }
I20250112 20:52:42.481143 9059 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee554c435fc4dc99d07d138915e7e8e. Latest consensus state: current_term: 1 leader_uuid: "dee554c435fc4dc99d07d138915e7e8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee554c435fc4dc99d07d138915e7e8e" member_type: VOTER } }
I20250112 20:52:42.481331 9058 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:42.481348 9059 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:42.482909 9062 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:42.484683 9062 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:42.485221 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:42.487608 9062 catalog_manager.cc:1348] Generated new cluster ID: b939b4e7a69b45a782ae202ca4ce6d4f
I20250112 20:52:42.487681 9062 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:42.515208 9062 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:42.515978 9062 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:42.524899 9062 catalog_manager.cc:5899] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e: Generated new TSK 0
I20250112 20:52:42.525120 9062 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:42.530440 9009 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57292:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:42.549971 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:42.552410 9081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:42.552505 9082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:42.553203 9084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:42.553628 416 server_base.cc:1034] running on GCE node
I20250112 20:52:42.553864 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:42.553911 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:42.553943 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715162553939 us; error 0 us; skew 500 ppm
I20250112 20:52:42.554198 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:42.555371 416 webserver.cc:458] Webserver started at http://127.0.104.1:37289/ using document root <none> and password file <none>
I20250112 20:52:42.555577 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:42.555634 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:42.555753 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:42.556486 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "606014f7b7054ee3bbfcca6df0512d2f"
format_stamp: "Formatted at 2025-01-12 20:52:42 on dist-test-slave-3gql"
I20250112 20:52:42.559248 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250112 20:52:42.560729 9089 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:42.561028 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250112 20:52:42.561111 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "606014f7b7054ee3bbfcca6df0512d2f"
format_stamp: "Formatted at 2025-01-12 20:52:42 on dist-test-slave-3gql"
I20250112 20:52:42.561228 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:42.575829 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:42.576344 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:42.578092 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:42.578166 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:42.578264 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:42.578308 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:42.593777 9158 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:36587 every 8 connection(s)
I20250112 20:52:42.593758 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:36587
I20250112 20:52:42.595181 9159 heartbeater.cc:346] Connected to a master server at 127.0.104.62:37885
I20250112 20:52:42.595276 9159 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:42.595541 9159 heartbeater.cc:510] Master 127.0.104.62:37885 requested a full tablet report, sending...
I20250112 20:52:42.596179 9009 ts_manager.cc:194] Registered new tserver with Master: 606014f7b7054ee3bbfcca6df0512d2f (127.0.104.1:36587)
I20250112 20:52:42.596872 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002261965s
I20250112 20:52:42.597271 9009 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57294
I20250112 20:52:42.605643 9009 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57306:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:42.616636 9122 tablet_service.cc:1467] Processing CreateTablet for tablet 80e461c16c504489ac40411bc25ec7a5 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=36030d33fe864a7eb38de6c0a644da01]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:42.616672 9123 tablet_service.cc:1467] Processing CreateTablet for tablet 0f0fca62dff74a46876314dc3d0e59c9 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=36030d33fe864a7eb38de6c0a644da01]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:42.617022 9122 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80e461c16c504489ac40411bc25ec7a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:42.617233 9123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f0fca62dff74a46876314dc3d0e59c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:42.621414 9169 tablet_bootstrap.cc:492] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f: Bootstrap starting.
I20250112 20:52:42.622502 9169 tablet_bootstrap.cc:654] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:42.623749 9169 tablet_bootstrap.cc:492] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f: No bootstrap required, opened a new log
I20250112 20:52:42.623850 9169 ts_tablet_manager.cc:1397] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:52:42.624351 9169 raft_consensus.cc:357] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.624473 9169 raft_consensus.cc:383] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:42.624562 9169 raft_consensus.cc:738] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606014f7b7054ee3bbfcca6df0512d2f, State: Initialized, Role: FOLLOWER
I20250112 20:52:42.624740 9169 consensus_queue.cc:260] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.624861 9169 raft_consensus.cc:397] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:42.624981 9169 raft_consensus.cc:491] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:42.625053 9169 raft_consensus.cc:3054] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:42.626631 9169 raft_consensus.cc:513] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.626801 9169 leader_election.cc:304] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606014f7b7054ee3bbfcca6df0512d2f; no voters:
I20250112 20:52:42.627157 9169 leader_election.cc:290] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:42.627295 9171 raft_consensus.cc:2798] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:42.627686 9159 heartbeater.cc:502] Master 127.0.104.62:37885 was elected leader, sending a full tablet report...
I20250112 20:52:42.627729 9169 ts_tablet_manager.cc:1428] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:42.627746 9171 raft_consensus.cc:695] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 LEADER]: Becoming Leader. State: Replica: 606014f7b7054ee3bbfcca6df0512d2f, State: Running, Role: LEADER
I20250112 20:52:42.627916 9169 tablet_bootstrap.cc:492] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f: Bootstrap starting.
I20250112 20:52:42.627987 9171 consensus_queue.cc:237] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.629403 9169 tablet_bootstrap.cc:654] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:42.630393 9009 catalog_manager.cc:5526] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 606014f7b7054ee3bbfcca6df0512d2f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "606014f7b7054ee3bbfcca6df0512d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:42.630772 9169 tablet_bootstrap.cc:492] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f: No bootstrap required, opened a new log
I20250112 20:52:42.630877 9169 ts_tablet_manager.cc:1397] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:42.631433 9169 raft_consensus.cc:357] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.631603 9169 raft_consensus.cc:383] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:42.631662 9169 raft_consensus.cc:738] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606014f7b7054ee3bbfcca6df0512d2f, State: Initialized, Role: FOLLOWER
I20250112 20:52:42.631834 9169 consensus_queue.cc:260] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.631989 9169 raft_consensus.cc:397] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:42.632055 9169 raft_consensus.cc:491] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:42.632128 9169 raft_consensus.cc:3054] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:42.633726 9169 raft_consensus.cc:513] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.633877 9169 leader_election.cc:304] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606014f7b7054ee3bbfcca6df0512d2f; no voters:
I20250112 20:52:42.633984 9169 leader_election.cc:290] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:42.634065 9171 raft_consensus.cc:2798] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:42.634320 9171 raft_consensus.cc:695] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 LEADER]: Becoming Leader. State: Replica: 606014f7b7054ee3bbfcca6df0512d2f, State: Running, Role: LEADER
I20250112 20:52:42.634501 9169 ts_tablet_manager.cc:1428] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:42.634572 9171 consensus_queue.cc:237] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:42.636766 9009 catalog_manager.cc:5526] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 606014f7b7054ee3bbfcca6df0512d2f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "606014f7b7054ee3bbfcca6df0512d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:42.647351 9165 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:43.542829 9009 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57316:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:43.552732 9123 tablet_service.cc:1467] Processing CreateTablet for tablet 3ae13b76504d4440a392123f089d74f9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5adabcba7f0487593b5bca1b18a8f34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:43.553115 9123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ae13b76504d4440a392123f089d74f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:43.557250 9187 tablet_bootstrap.cc:492] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: Bootstrap starting.
I20250112 20:52:43.558684 9187 tablet_bootstrap.cc:654] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:43.560115 9187 tablet_bootstrap.cc:492] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: No bootstrap required, opened a new log
I20250112 20:52:43.560256 9187 ts_tablet_manager.cc:1397] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:43.560837 9187 raft_consensus.cc:357] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:43.560978 9187 raft_consensus.cc:383] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:43.561038 9187 raft_consensus.cc:738] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 606014f7b7054ee3bbfcca6df0512d2f, State: Initialized, Role: FOLLOWER
I20250112 20:52:43.561251 9187 consensus_queue.cc:260] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:43.561414 9187 raft_consensus.cc:397] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:43.561482 9187 raft_consensus.cc:491] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:43.561554 9187 raft_consensus.cc:3054] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:43.563369 9187 raft_consensus.cc:513] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:43.563549 9187 leader_election.cc:304] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 606014f7b7054ee3bbfcca6df0512d2f; no voters:
I20250112 20:52:43.563951 9187 leader_election.cc:290] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:43.564113 9189 raft_consensus.cc:2798] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:43.564631 9187 ts_tablet_manager.cc:1428] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250112 20:52:43.564913 9189 raft_consensus.cc:695] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 LEADER]: Becoming Leader. State: Replica: 606014f7b7054ee3bbfcca6df0512d2f, State: Running, Role: LEADER
I20250112 20:52:43.565140 9189 consensus_queue.cc:237] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } }
I20250112 20:52:43.566136 9191 tablet_replica.cc:440] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: TxnStatusTablet state changed. Reason: New leader 606014f7b7054ee3bbfcca6df0512d2f. Latest consensus state: current_term: 1 leader_uuid: "606014f7b7054ee3bbfcca6df0512d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } } }
I20250112 20:52:43.566298 9191 tablet_replica.cc:443] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:43.566649 9190 tablet_replica.cc:440] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "606014f7b7054ee3bbfcca6df0512d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } } }
I20250112 20:52:43.566908 9190 tablet_replica.cc:443] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:43.567759 9009 catalog_manager.cc:5526] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 606014f7b7054ee3bbfcca6df0512d2f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "606014f7b7054ee3bbfcca6df0512d2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "606014f7b7054ee3bbfcca6df0512d2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36587 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:43.569129 9193 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:43.569244 9193 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250112 20:52:44.623445 9204 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
W20250112 20:52:44.623565 9204 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42132: Aborted: operation has been aborted
W20250112 20:52:44.624101 9164 batcher.cc:441] Aborted: Failed to write batch of 1 ops to tablet 0f0fca62dff74a46876314dc3d0e59c9 after 1 attempt(s): Failed to write to server: 606014f7b7054ee3bbfcca6df0512d2f (127.0.104.1:36587): operation has been aborted
I20250112 20:52:44.636274 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:44.644507 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:45.636624 416 tablet_replica.cc:331] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f: stopping tablet replica
W20250112 20:52:45.636646 9209 tablet_replica.cc:1307] Illegal state: Transaction metadata for transaction 0 already exists: cancelling pending write operations
I20250112 20:52:45.637009 416 raft_consensus.cc:2238] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:45.637219 416 raft_consensus.cc:2267] T 3ae13b76504d4440a392123f089d74f9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:45.637801 416 tablet_replica.cc:331] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f: stopping tablet replica
I20250112 20:52:45.637923 416 raft_consensus.cc:2238] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:45.638058 416 raft_consensus.cc:2267] T 0f0fca62dff74a46876314dc3d0e59c9 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:45.638479 416 tablet_replica.cc:331] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f: stopping tablet replica
I20250112 20:52:45.638597 416 raft_consensus.cc:2238] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:45.638715 416 raft_consensus.cc:2267] T 80e461c16c504489ac40411bc25ec7a5 P 606014f7b7054ee3bbfcca6df0512d2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:45.652945 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:45.657522 416 master.cc:537] Master@127.0.104.62:37885 shutting down...
I20250112 20:52:45.666851 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:45.667122 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:45.667243 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P dee554c435fc4dc99d07d138915e7e8e: stopping tablet replica
I20250112 20:52:45.680471 416 master.cc:559] Master@127.0.104.62:37885 shutdown complete.
[ OK ] TxnOpDispatcherITest.RollbackWriteOpPendingParticipantRegistered (3255 ms)
[ RUN ] TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted
I20250112 20:52:45.690824 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:39125
I20250112 20:52:45.691320 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:45.693548 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:45.693611 9210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:45.693985 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:45.694598 416 server_base.cc:1034] running on GCE node
I20250112 20:52:45.694813 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:45.694880 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:45.694947 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715165694917 us; error 0 us; skew 500 ppm
I20250112 20:52:45.695123 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:45.696664 416 webserver.cc:458] Webserver started at http://127.0.104.62:34575/ using document root <none> and password file <none>
I20250112 20:52:45.696858 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:45.696923 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:45.697036 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:45.697600 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "3a5af71cde4c40e09130ccbe7f24a7ff"
format_stamp: "Formatted at 2025-01-12 20:52:45 on dist-test-slave-3gql"
I20250112 20:52:45.699905 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:45.701301 9218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:45.701601 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:45.701710 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "3a5af71cde4c40e09130ccbe7f24a7ff"
format_stamp: "Formatted at 2025-01-12 20:52:45 on dist-test-slave-3gql"
I20250112 20:52:45.701820 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:45.723007 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:45.723629 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:45.736191 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:39125
I20250112 20:52:45.736202 9279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:39125 every 8 connection(s)
I20250112 20:52:45.737493 9280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:45.741410 9280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff: Bootstrap starting.
I20250112 20:52:45.742550 9280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:45.743876 9280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff: No bootstrap required, opened a new log
I20250112 20:52:45.744462 9280 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" member_type: VOTER }
I20250112 20:52:45.744661 9280 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:45.744735 9280 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5af71cde4c40e09130ccbe7f24a7ff, State: Initialized, Role: FOLLOWER
I20250112 20:52:45.744938 9280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" member_type: VOTER }
I20250112 20:52:45.745069 9280 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:45.745143 9280 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:45.745231 9280 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:45.746834 9280 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" member_type: VOTER }
I20250112 20:52:45.747035 9280 leader_election.cc:304] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a5af71cde4c40e09130ccbe7f24a7ff; no voters:
I20250112 20:52:45.747377 9280 leader_election.cc:290] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:45.747511 9283 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:45.747946 9283 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 1 LEADER]: Becoming Leader. State: Replica: 3a5af71cde4c40e09130ccbe7f24a7ff, State: Running, Role: LEADER
I20250112 20:52:45.748181 9280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:45.748163 9283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" member_type: VOTER }
I20250112 20:52:45.749086 9285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a5af71cde4c40e09130ccbe7f24a7ff. Latest consensus state: current_term: 1 leader_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" member_type: VOTER } }
I20250112 20:52:45.749040 9284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a5af71cde4c40e09130ccbe7f24a7ff" member_type: VOTER } }
I20250112 20:52:45.749312 9285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:45.749457 9284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:45.753459 9288 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:45.754680 9288 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:45.756214 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:45.758198 9288 catalog_manager.cc:1348] Generated new cluster ID: 330655be379d4363a8b7d0bb289b129d
I20250112 20:52:45.758291 9288 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:45.767789 9288 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:45.768782 9288 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:45.776065 9288 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff: Generated new TSK 0
I20250112 20:52:45.776298 9288 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:45.782789 9235 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48684:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:45.789003 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:45.791425 9307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:45.791782 9310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:45.792158 416 server_base.cc:1034] running on GCE node
W20250112 20:52:45.791782 9308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:45.792500 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:45.792593 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:45.792651 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715165792633 us; error 0 us; skew 500 ppm
I20250112 20:52:45.792805 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:45.793941 416 webserver.cc:458] Webserver started at http://127.0.104.1:42207/ using document root <none> and password file <none>
I20250112 20:52:45.794173 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:45.794252 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:45.794385 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:45.794973 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "b87459919d634513a4c585d5f7716461"
format_stamp: "Formatted at 2025-01-12 20:52:45 on dist-test-slave-3gql"
I20250112 20:52:45.797322 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250112 20:52:45.798696 9315 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:45.799062 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:45.799158 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "b87459919d634513a4c585d5f7716461"
format_stamp: "Formatted at 2025-01-12 20:52:45 on dist-test-slave-3gql"
I20250112 20:52:45.799268 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:45.812155 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:45.812722 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:45.814244 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:45.814409 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:45.814530 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:45.814610 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:45.830492 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:42245
I20250112 20:52:45.830498 9384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:42245 every 8 connection(s)
I20250112 20:52:45.832064 9385 heartbeater.cc:346] Connected to a master server at 127.0.104.62:39125
I20250112 20:52:45.832187 9385 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:45.832505 9385 heartbeater.cc:510] Master 127.0.104.62:39125 requested a full tablet report, sending...
I20250112 20:52:45.833305 9235 ts_manager.cc:194] Registered new tserver with Master: b87459919d634513a4c585d5f7716461 (127.0.104.1:42245)
I20250112 20:52:45.833763 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002322388s
I20250112 20:52:45.834519 9235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48696
I20250112 20:52:45.843501 9235 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48714:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:52:45.855413 9345 tablet_service.cc:1467] Processing CreateTablet for tablet 1a4946dc74d34712917baa9ea7983b3e (DEFAULT_TABLE table=txn_op_dispatcher_test [id=33db8f8bc57742c2bd46defd39bb075f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:45.855481 9344 tablet_service.cc:1467] Processing CreateTablet for tablet b3450051510c49db854227ab08dafc9e (DEFAULT_TABLE table=txn_op_dispatcher_test [id=33db8f8bc57742c2bd46defd39bb075f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:45.855846 9345 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a4946dc74d34712917baa9ea7983b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:45.856046 9344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3450051510c49db854227ab08dafc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:45.860460 9395 tablet_bootstrap.cc:492] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461: Bootstrap starting.
I20250112 20:52:45.861590 9395 tablet_bootstrap.cc:654] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:45.862867 9395 tablet_bootstrap.cc:492] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461: No bootstrap required, opened a new log
I20250112 20:52:45.863004 9395 ts_tablet_manager.cc:1397] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:45.863555 9395 raft_consensus.cc:357] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.863699 9395 raft_consensus.cc:383] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:45.863776 9395 raft_consensus.cc:738] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87459919d634513a4c585d5f7716461, State: Initialized, Role: FOLLOWER
I20250112 20:52:45.863955 9395 consensus_queue.cc:260] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.864089 9395 raft_consensus.cc:397] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:45.864159 9395 raft_consensus.cc:491] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:45.864243 9395 raft_consensus.cc:3054] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:45.865775 9395 raft_consensus.cc:513] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.865967 9395 leader_election.cc:304] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87459919d634513a4c585d5f7716461; no voters:
I20250112 20:52:45.866348 9395 leader_election.cc:290] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:45.866489 9397 raft_consensus.cc:2798] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:45.866986 9397 raft_consensus.cc:695] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 1 LEADER]: Becoming Leader. State: Replica: b87459919d634513a4c585d5f7716461, State: Running, Role: LEADER
I20250112 20:52:45.866991 9385 heartbeater.cc:502] Master 127.0.104.62:39125 was elected leader, sending a full tablet report...
I20250112 20:52:45.867051 9395 ts_tablet_manager.cc:1428] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:52:45.867478 9397 consensus_queue.cc:237] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.867604 9395 tablet_bootstrap.cc:492] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: Bootstrap starting.
I20250112 20:52:45.869020 9395 tablet_bootstrap.cc:654] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:45.870007 9235 catalog_manager.cc:5526] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87459919d634513a4c585d5f7716461 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b87459919d634513a4c585d5f7716461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:45.870486 9395 tablet_bootstrap.cc:492] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: No bootstrap required, opened a new log
I20250112 20:52:45.870637 9395 ts_tablet_manager.cc:1397] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:45.871265 9395 raft_consensus.cc:357] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.871443 9395 raft_consensus.cc:383] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:45.871529 9395 raft_consensus.cc:738] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87459919d634513a4c585d5f7716461, State: Initialized, Role: FOLLOWER
I20250112 20:52:45.871788 9395 consensus_queue.cc:260] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.871964 9395 raft_consensus.cc:397] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:45.872063 9395 raft_consensus.cc:491] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:45.872171 9395 raft_consensus.cc:3054] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:45.874500 9395 raft_consensus.cc:513] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.874696 9395 leader_election.cc:304] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87459919d634513a4c585d5f7716461; no voters:
I20250112 20:52:45.874855 9395 leader_election.cc:290] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:45.874963 9398 raft_consensus.cc:2798] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:45.875171 9398 raft_consensus.cc:695] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 1 LEADER]: Becoming Leader. State: Replica: b87459919d634513a4c585d5f7716461, State: Running, Role: LEADER
I20250112 20:52:45.875285 9395 ts_tablet_manager.cc:1428] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:52:45.875439 9398 consensus_queue.cc:237] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:45.878332 9235 catalog_manager.cc:5526] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87459919d634513a4c585d5f7716461 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b87459919d634513a4c585d5f7716461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } health_report { overall_health: HEALTHY } } }
W20250112 20:52:45.883270 9388 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:46.794553 9235 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48730:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:46.801385 9344 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4083a44d9e4f8ebf3a82d551487f17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fe26f7f905af447fac6e59efac58b043]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:46.801761 9344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c4083a44d9e4f8ebf3a82d551487f17. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:46.805903 9413 tablet_bootstrap.cc:492] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: Bootstrap starting.
I20250112 20:52:46.807322 9413 tablet_bootstrap.cc:654] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:46.808529 9413 tablet_bootstrap.cc:492] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: No bootstrap required, opened a new log
I20250112 20:52:46.808672 9413 ts_tablet_manager.cc:1397] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:52:46.809155 9413 raft_consensus.cc:357] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:46.809305 9413 raft_consensus.cc:383] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:46.809365 9413 raft_consensus.cc:738] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87459919d634513a4c585d5f7716461, State: Initialized, Role: FOLLOWER
I20250112 20:52:46.809535 9413 consensus_queue.cc:260] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:46.809664 9413 raft_consensus.cc:397] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:46.809718 9413 raft_consensus.cc:491] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:46.809790 9413 raft_consensus.cc:3054] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:46.811532 9413 raft_consensus.cc:513] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:46.811717 9413 leader_election.cc:304] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87459919d634513a4c585d5f7716461; no voters:
I20250112 20:52:46.812050 9413 leader_election.cc:290] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:46.812155 9415 raft_consensus.cc:2798] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:46.812508 9415 raft_consensus.cc:695] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 1 LEADER]: Becoming Leader. State: Replica: b87459919d634513a4c585d5f7716461, State: Running, Role: LEADER
I20250112 20:52:46.812780 9415 consensus_queue.cc:237] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } }
I20250112 20:52:46.813480 9415 tablet_replica.cc:440] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: TxnStatusTablet state changed. Reason: New leader b87459919d634513a4c585d5f7716461. Latest consensus state: current_term: 1 leader_uuid: "b87459919d634513a4c585d5f7716461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } } }
I20250112 20:52:46.813851 9417 tablet_replica.cc:440] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b87459919d634513a4c585d5f7716461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } } }
I20250112 20:52:46.814009 9417 tablet_replica.cc:443] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:46.814499 9415 tablet_replica.cc:443] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:46.814931 9234 catalog_manager.cc:5526] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 reported cstate change: term changed from 0 to 1, leader changed from <none> to b87459919d634513a4c585d5f7716461 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b87459919d634513a4c585d5f7716461" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87459919d634513a4c585d5f7716461" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 42245 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:46.818459 9419 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:46.818657 9419 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:46.818758 9413 ts_tablet_manager.cc:1428] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: Time spent starting tablet: real 0.010s user 0.000s sys 0.010s
I20250112 20:52:47.946244 9424 tablet_replica.cc:331] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: stopping tablet replica
I20250112 20:52:47.946480 9424 raft_consensus.cc:2238] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:47.946676 9424 raft_consensus.cc:2267] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:47.947371 9424 ts_tablet_manager.cc:1905] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250112 20:52:47.951689 9424 ts_tablet_manager.cc:1918] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250112 20:52:47.951812 9424 log.cc:1198] T 1a4946dc74d34712917baa9ea7983b3e P b87459919d634513a4c585d5f7716461: Deleting WAL directory at /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0/minicluster-data/ts-0-root/wals/1a4946dc74d34712917baa9ea7983b3e
W20250112 20:52:48.962323 9427 tablet_replica.cc:1307] Illegal state: The tablet is not in a running state: STOPPED: cancelling pending write operations
W20250112 20:52:48.962492 9427 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:53458: Illegal state: The tablet is not in a running state: STOPPED
I20250112 20:52:48.962747 9427 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:53458 (ReqId={client: 687c097cec2b49c7841ce21567f40caf, seq_no=0, attempt_no=0}) took 2010 ms. Trace:
I20250112 20:52:48.962821 9427 rpcz_store.cc:276] 0112 20:52:46.951985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:46.952057 (+ 72us) service_pool.cc:225] Handling call
0112 20:52:48.962725 (+2010668us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":231,"threads_started":1,"txn-participant-registration.queue_time_us":319}
W20250112 20:52:56.947816 9388 meta_cache.cc:1015] Timed out: LookupRpc { table: 'txn_op_dispatcher_test', partition-key: (HASH (key): 0, RANGE (key): -9223372036854775808), attempt: 1 } failed: LookupRpc timed out after deadline expired: GetTableLocations RPC to 127.0.104.62:39125 timed out after 0.000s (SENT)
W20250112 20:52:56.947965 9388 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet 1a4946dc74d34712917baa9ea7983b3e after 122 attempt(s): LookupRpc { table: 'txn_op_dispatcher_test', partition-key: (HASH (key): 0, RANGE (key): -9223372036854775808), attempt: 1 } failed: LookupRpc timed out after deadline expired: GetTableLocations RPC to 127.0.104.62:39125 timed out after 0.000s (SENT)
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/txn_write_ops-itest.cc:2052: Failure
Value of: op_status.ToString()
Expected: has substring "STOPPED"
Actual: "Timed out: Failed to write batch of 1 ops to tablet 1a4946dc74d34712917baa9ea7983b3e after 122 attempt(s): LookupRpc { table: 'txn_op_dispatcher_test', partition-key: (HASH (key): 0, RANGE (key): -9223372036854775808), attempt: 1 } failed: LookupRpc timed out after deadline expired: GetTableLocations RPC to 127.0.104.62:39125 timed out after 0.000s (SENT)"
I20250112 20:52:56.949075 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:52:56.964329 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:52:56.964653 416 tablet_replica.cc:331] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461: stopping tablet replica
I20250112 20:52:56.964841 416 raft_consensus.cc:2238] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:56.965039 416 raft_consensus.cc:2267] T 3c4083a44d9e4f8ebf3a82d551487f17 P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:56.965739 416 tablet_replica.cc:331] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461: stopping tablet replica
I20250112 20:52:56.965864 416 raft_consensus.cc:2238] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:56.966001 416 raft_consensus.cc:2267] T b3450051510c49db854227ab08dafc9e P b87459919d634513a4c585d5f7716461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:56.980197 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:52:56.984886 416 master.cc:537] Master@127.0.104.62:39125 shutting down...
I20250112 20:52:56.994617 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:56.994868 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:56.994967 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3a5af71cde4c40e09130ccbe7f24a7ff: stopping tablet replica
I20250112 20:52:57.008318 416 master.cc:559] Master@127.0.104.62:39125 shutdown complete.
I20250112 20:52:57.014577 416 test_util.cc:181] -----------------------------------------------
I20250112 20:52:57.014667 416 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted.1736715060603202-416-0
[ FAILED ] TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted (11325 ms)
[ RUN ] TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown
I20250112 20:52:57.016753 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:35943
I20250112 20:52:57.017277 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:57.019704 9430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:57.019884 9431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:57.020114 9433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:57.020879 416 server_base.cc:1034] running on GCE node
I20250112 20:52:57.021111 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:57.021193 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:57.021234 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715177021220 us; error 0 us; skew 500 ppm
I20250112 20:52:57.021381 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:57.028412 416 webserver.cc:458] Webserver started at http://127.0.104.62:35533/ using document root <none> and password file <none>
I20250112 20:52:57.028700 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:57.028798 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:57.028919 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:57.029561 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "f38fc768f79943f493d4e782192f24e3"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.032065 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:57.033726 9438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.034082 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250112 20:52:57.034206 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "f38fc768f79943f493d4e782192f24e3"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.034332 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:57.043051 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:57.043668 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:57.055975 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35943
I20250112 20:52:57.055995 9499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35943 every 8 connection(s)
I20250112 20:52:57.057291 9500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:57.061175 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3: Bootstrap starting.
I20250112 20:52:57.062309 9500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:57.063697 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3: No bootstrap required, opened a new log
I20250112 20:52:57.064309 9500 raft_consensus.cc:357] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38fc768f79943f493d4e782192f24e3" member_type: VOTER }
I20250112 20:52:57.064442 9500 raft_consensus.cc:383] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:57.064502 9500 raft_consensus.cc:738] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f38fc768f79943f493d4e782192f24e3, State: Initialized, Role: FOLLOWER
I20250112 20:52:57.064721 9500 consensus_queue.cc:260] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38fc768f79943f493d4e782192f24e3" member_type: VOTER }
I20250112 20:52:57.064841 9500 raft_consensus.cc:397] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:52:57.064896 9500 raft_consensus.cc:491] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:52:57.064963 9500 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:57.066579 9500 raft_consensus.cc:513] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38fc768f79943f493d4e782192f24e3" member_type: VOTER }
I20250112 20:52:57.066759 9500 leader_election.cc:304] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f38fc768f79943f493d4e782192f24e3; no voters:
I20250112 20:52:57.067099 9500 leader_election.cc:290] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:52:57.067241 9503 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:57.067687 9503 raft_consensus.cc:695] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 1 LEADER]: Becoming Leader. State: Replica: f38fc768f79943f493d4e782192f24e3, State: Running, Role: LEADER
I20250112 20:52:57.067970 9503 consensus_queue.cc:237] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38fc768f79943f493d4e782192f24e3" member_type: VOTER }
I20250112 20:52:57.068032 9500 sys_catalog.cc:564] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:52:57.068895 9504 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f38fc768f79943f493d4e782192f24e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38fc768f79943f493d4e782192f24e3" member_type: VOTER } }
I20250112 20:52:57.068955 9505 sys_catalog.cc:455] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f38fc768f79943f493d4e782192f24e3. Latest consensus state: current_term: 1 leader_uuid: "f38fc768f79943f493d4e782192f24e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f38fc768f79943f493d4e782192f24e3" member_type: VOTER } }
I20250112 20:52:57.069219 9505 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:57.069195 9504 sys_catalog.cc:458] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [sys.catalog]: This master's current role is: LEADER
I20250112 20:52:57.073216 9508 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:52:57.073459 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:52:57.074931 9508 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:52:57.078548 9508 catalog_manager.cc:1348] Generated new cluster ID: 2ec9e81536f240aaa0c98e91d9fe350d
I20250112 20:52:57.078642 9508 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:52:57.100844 9508 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:52:57.101677 9508 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:52:57.109007 9508 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3: Generated new TSK 0
I20250112 20:52:57.109284 9508 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:52:57.120997 9455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34658:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:52:57.138571 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:52:57.141214 9527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:57.141310 9528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:57.141908 9530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:57.142230 416 server_base.cc:1034] running on GCE node
I20250112 20:52:57.142460 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:57.142524 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:57.142566 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715177142552 us; error 0 us; skew 500 ppm
I20250112 20:52:57.142690 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:57.143910 416 webserver.cc:458] Webserver started at http://127.0.104.1:41131/ using document root <none> and password file <none>
I20250112 20:52:57.144135 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:57.144208 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:57.144317 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:57.145012 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "2d99bb94e30148eebc368262a099475c"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.147583 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250112 20:52:57.149154 9535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.149504 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250112 20:52:57.149617 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "2d99bb94e30148eebc368262a099475c"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.149751 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:57.158185 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:57.158768 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:57.160472 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:57.160672 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.160794 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:57.160880 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.186861 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46379
I20250112 20:52:57.186897 9604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46379 every 8 connection(s)
I20250112 20:52:57.188336 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 20:52:57.188505 9605 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35943
I20250112 20:52:57.188664 9605 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:57.189039 9605 heartbeater.cc:510] Master 127.0.104.62:35943 requested a full tablet report, sending...
I20250112 20:52:57.189947 9455 ts_manager.cc:194] Registered new tserver with Master: 2d99bb94e30148eebc368262a099475c (127.0.104.1:46379)
W20250112 20:52:57.191005 9608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:57.191089 9607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:57.191395 9455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34660
W20250112 20:52:57.194069 9610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:57.194121 416 server_base.cc:1034] running on GCE node
I20250112 20:52:57.194444 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:57.194520 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:57.194583 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715177194563 us; error 0 us; skew 500 ppm
I20250112 20:52:57.194733 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:57.195981 416 webserver.cc:458] Webserver started at http://127.0.104.2:45679/ using document root <none> and password file <none>
I20250112 20:52:57.196214 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:57.196301 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:57.196435 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:57.197072 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-1-root/instance:
uuid: "774fad6247404200a05df7e1699c3e7b"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.199440 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:52:57.201143 9615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.201511 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250112 20:52:57.201627 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-1-root
uuid: "774fad6247404200a05df7e1699c3e7b"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.201748 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:57.212792 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:57.213379 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:57.214994 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:57.215098 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.215222 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:57.215291 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.241365 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33601
I20250112 20:52:57.241365 9684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33601 every 8 connection(s)
I20250112 20:52:57.243185 9685 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35943
I20250112 20:52:57.243304 9685 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:57.243564 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 20:52:57.243681 9685 heartbeater.cc:510] Master 127.0.104.62:35943 requested a full tablet report, sending...
I20250112 20:52:57.244589 9455 ts_manager.cc:194] Registered new tserver with Master: 774fad6247404200a05df7e1699c3e7b (127.0.104.2:33601)
I20250112 20:52:57.245731 9455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34676
W20250112 20:52:57.246310 9687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:52:57.246315 9688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:57.249521 416 server_base.cc:1034] running on GCE node
W20250112 20:52:57.249650 9690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:52:57.249979 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:52:57.250036 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:52:57.250064 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715177250060 us; error 0 us; skew 500 ppm
I20250112 20:52:57.250203 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:52:57.251412 416 webserver.cc:458] Webserver started at http://127.0.104.3:37163/ using document root <none> and password file <none>
I20250112 20:52:57.251617 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:52:57.251677 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:52:57.251765 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:52:57.252331 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-2-root/instance:
uuid: "60f39687013e4b65afd587e9d08f2d95"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.254657 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250112 20:52:57.256103 9695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.256531 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:52:57.256652 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-2-root
uuid: "60f39687013e4b65afd587e9d08f2d95"
format_stamp: "Formatted at 2025-01-12 20:52:57 on dist-test-slave-3gql"
I20250112 20:52:57.256762 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown.1736715060603202-416-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:52:57.273026 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:52:57.273540 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:52:57.275117 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:52:57.275242 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.275375 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:52:57.275439 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:52:57.296864 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:41963
I20250112 20:52:57.296895 9764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:41963 every 8 connection(s)
I20250112 20:52:57.298615 9765 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35943
I20250112 20:52:57.298731 9765 heartbeater.cc:463] Registering TS with master...
I20250112 20:52:57.299093 9765 heartbeater.cc:510] Master 127.0.104.62:35943 requested a full tablet report, sending...
I20250112 20:52:57.299872 9455 ts_manager.cc:194] Registered new tserver with Master: 60f39687013e4b65afd587e9d08f2d95 (127.0.104.3:41963)
I20250112 20:52:57.300133 416 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.002308967s
I20250112 20:52:57.300663 9455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34682
I20250112 20:52:57.310941 9455 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34704:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
W20250112 20:52:57.311569 9455 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table txn_op_dispatcher_test in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:52:57.326550 9566 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3b86f728844d09f49148c15dc8500 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=86607796a630481ab17e853222083765]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:57.326550 9564 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee36acc0f7c473db698156a7803e1d2 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=86607796a630481ab17e853222083765]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:57.327090 9566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0a3b86f728844d09f49148c15dc8500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:57.328169 9564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee36acc0f7c473db698156a7803e1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:57.328569 9729 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3b86f728844d09f49148c15dc8500 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=86607796a630481ab17e853222083765]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:57.328964 9729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0a3b86f728844d09f49148c15dc8500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:57.329316 9645 tablet_service.cc:1467] Processing CreateTablet for tablet e0a3b86f728844d09f49148c15dc8500 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=86607796a630481ab17e853222083765]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:57.329732 9645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0a3b86f728844d09f49148c15dc8500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:57.333164 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee36acc0f7c473db698156a7803e1d2 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=86607796a630481ab17e853222083765]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:57.333688 9644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee36acc0f7c473db698156a7803e1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:57.334892 9727 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee36acc0f7c473db698156a7803e1d2 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=86607796a630481ab17e853222083765]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:52:57.335350 9727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee36acc0f7c473db698156a7803e1d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:57.336192 9779 tablet_bootstrap.cc:492] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c: Bootstrap starting.
I20250112 20:52:57.338691 9779 tablet_bootstrap.cc:654] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:57.340009 9781 tablet_bootstrap.cc:492] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b: Bootstrap starting.
I20250112 20:52:57.343220 9781 tablet_bootstrap.cc:654] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:57.345064 9780 tablet_bootstrap.cc:492] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95: Bootstrap starting.
I20250112 20:52:57.345449 9781 tablet_bootstrap.cc:492] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b: No bootstrap required, opened a new log
I20250112 20:52:57.345638 9781 ts_tablet_manager.cc:1397] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250112 20:52:57.346472 9779 tablet_bootstrap.cc:492] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c: No bootstrap required, opened a new log
I20250112 20:52:57.346639 9779 ts_tablet_manager.cc:1397] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20250112 20:52:57.346580 9781 raft_consensus.cc:357] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.346848 9781 raft_consensus.cc:383] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:57.346956 9781 raft_consensus.cc:738] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774fad6247404200a05df7e1699c3e7b, State: Initialized, Role: FOLLOWER
I20250112 20:52:57.346988 9780 tablet_bootstrap.cc:654] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:57.347272 9781 consensus_queue.cc:260] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.347383 9779 raft_consensus.cc:357] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.347541 9779 raft_consensus.cc:383] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:57.347640 9779 raft_consensus.cc:738] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d99bb94e30148eebc368262a099475c, State: Initialized, Role: FOLLOWER
I20250112 20:52:57.347958 9779 consensus_queue.cc:260] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.348114 9685 heartbeater.cc:502] Master 127.0.104.62:35943 was elected leader, sending a full tablet report...
I20250112 20:52:57.348263 9781 ts_tablet_manager.cc:1428] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:52:57.349025 9781 tablet_bootstrap.cc:492] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b: Bootstrap starting.
I20250112 20:52:57.354048 9781 tablet_bootstrap.cc:654] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:57.354727 9779 ts_tablet_manager.cc:1428] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20250112 20:52:57.355194 9780 tablet_bootstrap.cc:492] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95: No bootstrap required, opened a new log
I20250112 20:52:57.355369 9780 ts_tablet_manager.cc:1397] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250112 20:52:57.355736 9605 heartbeater.cc:502] Master 127.0.104.62:35943 was elected leader, sending a full tablet report...
I20250112 20:52:57.355813 9779 tablet_bootstrap.cc:492] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c: Bootstrap starting.
I20250112 20:52:57.356068 9781 tablet_bootstrap.cc:492] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b: No bootstrap required, opened a new log
I20250112 20:52:57.356201 9781 ts_tablet_manager.cc:1397] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250112 20:52:57.357066 9780 raft_consensus.cc:357] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.357039 9781 raft_consensus.cc:357] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.357309 9780 raft_consensus.cc:383] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:57.357344 9781 raft_consensus.cc:383] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:57.357440 9780 raft_consensus.cc:738] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f39687013e4b65afd587e9d08f2d95, State: Initialized, Role: FOLLOWER
I20250112 20:52:57.357468 9781 raft_consensus.cc:738] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774fad6247404200a05df7e1699c3e7b, State: Initialized, Role: FOLLOWER
I20250112 20:52:57.357507 9779 tablet_bootstrap.cc:654] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:57.357775 9780 consensus_queue.cc:260] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.357774 9781 consensus_queue.cc:260] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.360177 9781 ts_tablet_manager.cc:1428] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250112 20:52:57.361642 9765 heartbeater.cc:502] Master 127.0.104.62:35943 was elected leader, sending a full tablet report...
I20250112 20:52:57.361814 9779 tablet_bootstrap.cc:492] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c: No bootstrap required, opened a new log
I20250112 20:52:57.361980 9779 ts_tablet_manager.cc:1397] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250112 20:52:57.362428 9780 ts_tablet_manager.cc:1428] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20250112 20:52:57.362769 9780 tablet_bootstrap.cc:492] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95: Bootstrap starting.
I20250112 20:52:57.362856 9779 raft_consensus.cc:357] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.363098 9779 raft_consensus.cc:383] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:57.363225 9779 raft_consensus.cc:738] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d99bb94e30148eebc368262a099475c, State: Initialized, Role: FOLLOWER
I20250112 20:52:57.363495 9779 consensus_queue.cc:260] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.364022 9779 ts_tablet_manager.cc:1428] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250112 20:52:57.365422 9780 tablet_bootstrap.cc:654] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:57.367246 9780 tablet_bootstrap.cc:492] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95: No bootstrap required, opened a new log
I20250112 20:52:57.367393 9780 ts_tablet_manager.cc:1397] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250112 20:52:57.368065 9780 raft_consensus.cc:357] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.368260 9780 raft_consensus.cc:383] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:57.368350 9780 raft_consensus.cc:738] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f39687013e4b65afd587e9d08f2d95, State: Initialized, Role: FOLLOWER
I20250112 20:52:57.368618 9780 consensus_queue.cc:260] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.369115 9780 ts_tablet_manager.cc:1428] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250112 20:52:57.415422 9786 raft_consensus.cc:491] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:52:57.415634 9786 raft_consensus.cc:513] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.417040 9786 leader_election.cc:290] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 774fad6247404200a05df7e1699c3e7b (127.0.104.2:33601), 60f39687013e4b65afd587e9d08f2d95 (127.0.104.3:41963)
I20250112 20:52:57.423084 9660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee36acc0f7c473db698156a7803e1d2" candidate_uuid: "2d99bb94e30148eebc368262a099475c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774fad6247404200a05df7e1699c3e7b" is_pre_election: true
I20250112 20:52:57.423350 9740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee36acc0f7c473db698156a7803e1d2" candidate_uuid: "2d99bb94e30148eebc368262a099475c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f39687013e4b65afd587e9d08f2d95" is_pre_election: true
I20250112 20:52:57.423560 9660 raft_consensus.cc:2463] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d99bb94e30148eebc368262a099475c in term 0.
I20250112 20:52:57.423609 9740 raft_consensus.cc:2463] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d99bb94e30148eebc368262a099475c in term 0.
I20250112 20:52:57.424086 9538 leader_election.cc:304] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d99bb94e30148eebc368262a099475c, 774fad6247404200a05df7e1699c3e7b; no voters:
I20250112 20:52:57.424386 9786 raft_consensus.cc:2798] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:52:57.424471 9786 raft_consensus.cc:491] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:52:57.424585 9786 raft_consensus.cc:3054] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:57.426617 9786 raft_consensus.cc:513] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.427238 9786 leader_election.cc:290] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [CANDIDATE]: Term 1 election: Requested vote from peers 774fad6247404200a05df7e1699c3e7b (127.0.104.2:33601), 60f39687013e4b65afd587e9d08f2d95 (127.0.104.3:41963)
I20250112 20:52:57.427577 9660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee36acc0f7c473db698156a7803e1d2" candidate_uuid: "2d99bb94e30148eebc368262a099475c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774fad6247404200a05df7e1699c3e7b"
I20250112 20:52:57.427596 9740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ee36acc0f7c473db698156a7803e1d2" candidate_uuid: "2d99bb94e30148eebc368262a099475c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f39687013e4b65afd587e9d08f2d95"
I20250112 20:52:57.427795 9660 raft_consensus.cc:3054] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:57.427822 9740 raft_consensus.cc:3054] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:57.430814 9660 raft_consensus.cc:2463] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d99bb94e30148eebc368262a099475c in term 1.
I20250112 20:52:57.430814 9740 raft_consensus.cc:2463] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d99bb94e30148eebc368262a099475c in term 1.
I20250112 20:52:57.431386 9538 leader_election.cc:304] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d99bb94e30148eebc368262a099475c, 774fad6247404200a05df7e1699c3e7b; no voters:
I20250112 20:52:57.431675 9786 raft_consensus.cc:2798] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:57.432096 9786 raft_consensus.cc:695] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 1 LEADER]: Becoming Leader. State: Replica: 2d99bb94e30148eebc368262a099475c, State: Running, Role: LEADER
I20250112 20:52:57.432384 9786 consensus_queue.cc:237] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.435176 9454 catalog_manager.cc:5526] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d99bb94e30148eebc368262a099475c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "2d99bb94e30148eebc368262a099475c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } health_report { overall_health: UNKNOWN } } }
W20250112 20:52:57.438071 9606 tablet.cc:2367] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250112 20:52:57.507263 9784 raft_consensus.cc:491] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:52:57.507510 9784 raft_consensus.cc:513] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.508401 9784 leader_election.cc:290] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d99bb94e30148eebc368262a099475c (127.0.104.1:46379), 60f39687013e4b65afd587e9d08f2d95 (127.0.104.3:41963)
I20250112 20:52:57.513682 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b86f728844d09f49148c15dc8500" candidate_uuid: "774fad6247404200a05df7e1699c3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d99bb94e30148eebc368262a099475c" is_pre_election: true
I20250112 20:52:57.513919 9580 raft_consensus.cc:2463] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774fad6247404200a05df7e1699c3e7b in term 0.
I20250112 20:52:57.514385 9740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b86f728844d09f49148c15dc8500" candidate_uuid: "774fad6247404200a05df7e1699c3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f39687013e4b65afd587e9d08f2d95" is_pre_election: true
I20250112 20:52:57.514421 9616 leader_election.cc:304] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d99bb94e30148eebc368262a099475c, 774fad6247404200a05df7e1699c3e7b; no voters:
I20250112 20:52:57.514619 9740 raft_consensus.cc:2463] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 774fad6247404200a05df7e1699c3e7b in term 0.
I20250112 20:52:57.514719 9784 raft_consensus.cc:2798] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:52:57.514832 9784 raft_consensus.cc:491] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:52:57.514940 9784 raft_consensus.cc:3054] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:57.516896 9784 raft_consensus.cc:513] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.517535 9784 leader_election.cc:290] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [CANDIDATE]: Term 1 election: Requested vote from peers 2d99bb94e30148eebc368262a099475c (127.0.104.1:46379), 60f39687013e4b65afd587e9d08f2d95 (127.0.104.3:41963)
I20250112 20:52:57.517885 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b86f728844d09f49148c15dc8500" candidate_uuid: "774fad6247404200a05df7e1699c3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d99bb94e30148eebc368262a099475c"
I20250112 20:52:57.518044 9580 raft_consensus.cc:3054] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:57.518014 9740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0a3b86f728844d09f49148c15dc8500" candidate_uuid: "774fad6247404200a05df7e1699c3e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60f39687013e4b65afd587e9d08f2d95"
I20250112 20:52:57.518215 9740 raft_consensus.cc:3054] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:57.520093 9740 raft_consensus.cc:2463] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774fad6247404200a05df7e1699c3e7b in term 1.
I20250112 20:52:57.520100 9580 raft_consensus.cc:2463] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 774fad6247404200a05df7e1699c3e7b in term 1.
I20250112 20:52:57.520536 9618 leader_election.cc:304] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d99bb94e30148eebc368262a099475c, 60f39687013e4b65afd587e9d08f2d95, 774fad6247404200a05df7e1699c3e7b; no voters:
I20250112 20:52:57.520879 9784 raft_consensus.cc:2798] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:57.521298 9784 raft_consensus.cc:695] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 1 LEADER]: Becoming Leader. State: Replica: 774fad6247404200a05df7e1699c3e7b, State: Running, Role: LEADER
I20250112 20:52:57.521615 9784 consensus_queue.cc:237] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:57.524067 9454 catalog_manager.cc:5526] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to 774fad6247404200a05df7e1699c3e7b (127.0.104.2). New cstate: current_term: 1 leader_uuid: "774fad6247404200a05df7e1699c3e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } health_report { overall_health: UNKNOWN } } }
W20250112 20:52:57.566488 9771 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:52:57.937317 9784 consensus_queue.cc:1035] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:52:57.942234 9784 consensus_queue.cc:1035] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:52:57.967031 9797 consensus_queue.cc:1035] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [LEADER]: Connected to new peer: Peer: permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:52:57.971204 9805 consensus_queue.cc:1035] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [LEADER]: Connected to new peer: Peer: permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:52:58.133422 9454 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34720:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:52:58.134155 9454 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250112 20:52:58.141845 9566 tablet_service.cc:1467] Processing CreateTablet for tablet 7046158d21c84cfe851c0fdf53de71d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e126ab9fe2694bbd91190c15e53b7856]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:58.142326 9566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7046158d21c84cfe851c0fdf53de71d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:58.142253 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 7046158d21c84cfe851c0fdf53de71d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e126ab9fe2694bbd91190c15e53b7856]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:58.142294 9729 tablet_service.cc:1467] Processing CreateTablet for tablet 7046158d21c84cfe851c0fdf53de71d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e126ab9fe2694bbd91190c15e53b7856]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:52:58.142800 9729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7046158d21c84cfe851c0fdf53de71d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:58.142799 9644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7046158d21c84cfe851c0fdf53de71d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:52:58.149145 9817 tablet_bootstrap.cc:492] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: Bootstrap starting.
I20250112 20:52:58.151021 9817 tablet_bootstrap.cc:654] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:58.153105 9819 tablet_bootstrap.cc:492] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: Bootstrap starting.
I20250112 20:52:58.153190 9817 tablet_bootstrap.cc:492] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: No bootstrap required, opened a new log
I20250112 20:52:58.153434 9817 ts_tablet_manager.cc:1397] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:52:58.154640 9819 tablet_bootstrap.cc:654] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:58.154676 9817 raft_consensus.cc:357] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.155026 9817 raft_consensus.cc:383] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:58.155159 9817 raft_consensus.cc:738] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d99bb94e30148eebc368262a099475c, State: Initialized, Role: FOLLOWER
I20250112 20:52:58.155508 9817 consensus_queue.cc:260] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.155993 9797 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.156268 9817 ts_tablet_manager.cc:1428] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250112 20:52:58.157358 9818 tablet_bootstrap.cc:492] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: Bootstrap starting.
I20250112 20:52:58.156304 9797 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:52:58.159690 9818 tablet_bootstrap.cc:654] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: Neither blocks nor log segments found. Creating new log.
I20250112 20:52:58.161955 9818 tablet_bootstrap.cc:492] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: No bootstrap required, opened a new log
I20250112 20:52:58.162178 9818 ts_tablet_manager.cc:1397] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250112 20:52:58.163072 9818 raft_consensus.cc:357] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.163329 9818 raft_consensus.cc:383] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:58.163451 9818 raft_consensus.cc:738] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774fad6247404200a05df7e1699c3e7b, State: Initialized, Role: FOLLOWER
I20250112 20:52:58.163611 9819 tablet_bootstrap.cc:492] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: No bootstrap required, opened a new log
I20250112 20:52:58.163763 9819 ts_tablet_manager.cc:1397] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20250112 20:52:58.163743 9818 consensus_queue.cc:260] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.164182 9795 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.164283 9818 ts_tablet_manager.cc:1428] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250112 20:52:58.164413 9795 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:52:58.165158 9819 raft_consensus.cc:357] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.165307 9819 raft_consensus.cc:383] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:52:58.165386 9819 raft_consensus.cc:738] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f39687013e4b65afd587e9d08f2d95, State: Initialized, Role: FOLLOWER
I20250112 20:52:58.165587 9819 consensus_queue.cc:260] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.166041 9819 ts_tablet_manager.cc:1428] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250112 20:52:58.165944 9801 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.166306 9801 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:52:58.330072 9801 raft_consensus.cc:491] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250112 20:52:58.330276 9801 raft_consensus.cc:513] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.331097 9801 leader_election.cc:290] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d99bb94e30148eebc368262a099475c (127.0.104.1:46379), 774fad6247404200a05df7e1699c3e7b (127.0.104.2:33601)
I20250112 20:52:58.340044 9660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046158d21c84cfe851c0fdf53de71d5" candidate_uuid: "60f39687013e4b65afd587e9d08f2d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774fad6247404200a05df7e1699c3e7b" is_pre_election: true
I20250112 20:52:58.340354 9660 raft_consensus.cc:2463] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60f39687013e4b65afd587e9d08f2d95 in term 0.
I20250112 20:52:58.340790 9698 leader_election.cc:304] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60f39687013e4b65afd587e9d08f2d95, 774fad6247404200a05df7e1699c3e7b; no voters:
I20250112 20:52:58.341128 9801 raft_consensus.cc:2798] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250112 20:52:58.341250 9801 raft_consensus.cc:491] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250112 20:52:58.341352 9801 raft_consensus.cc:3054] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:58.342764 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046158d21c84cfe851c0fdf53de71d5" candidate_uuid: "60f39687013e4b65afd587e9d08f2d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d99bb94e30148eebc368262a099475c" is_pre_election: true
I20250112 20:52:58.342985 9580 raft_consensus.cc:2463] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60f39687013e4b65afd587e9d08f2d95 in term 0.
I20250112 20:52:58.343672 9801 raft_consensus.cc:513] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.344206 9801 leader_election.cc:290] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [CANDIDATE]: Term 1 election: Requested vote from peers 2d99bb94e30148eebc368262a099475c (127.0.104.1:46379), 774fad6247404200a05df7e1699c3e7b (127.0.104.2:33601)
I20250112 20:52:58.344636 9660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046158d21c84cfe851c0fdf53de71d5" candidate_uuid: "60f39687013e4b65afd587e9d08f2d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774fad6247404200a05df7e1699c3e7b"
I20250112 20:52:58.344744 9580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7046158d21c84cfe851c0fdf53de71d5" candidate_uuid: "60f39687013e4b65afd587e9d08f2d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d99bb94e30148eebc368262a099475c"
I20250112 20:52:58.344846 9660 raft_consensus.cc:3054] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:58.344965 9580 raft_consensus.cc:3054] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:52:58.346731 9660 raft_consensus.cc:2463] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60f39687013e4b65afd587e9d08f2d95 in term 1.
I20250112 20:52:58.346735 9580 raft_consensus.cc:2463] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60f39687013e4b65afd587e9d08f2d95 in term 1.
I20250112 20:52:58.347118 9696 leader_election.cc:304] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d99bb94e30148eebc368262a099475c, 60f39687013e4b65afd587e9d08f2d95; no voters:
I20250112 20:52:58.347442 9801 raft_consensus.cc:2798] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:52:58.347908 9801 raft_consensus.cc:695] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 1 LEADER]: Becoming Leader. State: Replica: 60f39687013e4b65afd587e9d08f2d95, State: Running, Role: LEADER
I20250112 20:52:58.348255 9801 consensus_queue.cc:237] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } }
I20250112 20:52:58.349670 9827 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: TxnStatusTablet state changed. Reason: New leader 60f39687013e4b65afd587e9d08f2d95. Latest consensus state: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.349918 9827 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:58.350528 9829 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:52:58.350741 9829 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:52:58.351190 9455 catalog_manager.cc:5526] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60f39687013e4b65afd587e9d08f2d95 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } health_report { overall_health: HEALTHY } } }
I20250112 20:52:58.529138 9660 raft_consensus.cc:1270] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Refusing update from remote peer 60f39687013e4b65afd587e9d08f2d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:52:58.529157 9580 raft_consensus.cc:1270] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Refusing update from remote peer 60f39687013e4b65afd587e9d08f2d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250112 20:52:58.529660 9827 consensus_queue.cc:1035] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:52:58.529844 9801 consensus_queue.cc:1035] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250112 20:52:58.532459 9831 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: TxnStatusTablet state changed. Reason: New leader 60f39687013e4b65afd587e9d08f2d95. Latest consensus state: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.532713 9831 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:52:58.533887 9836 mvcc.cc:204] Tried to move back new op lower bound from 7113585371251003392 to 7113585370517508096. Current Snapshot: MvccSnapshot[applied={T|T < 7113585371251003392}]
I20250112 20:52:58.536228 9801 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.536477 9801 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:58.541314 9833 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: TxnStatusTablet state changed. Reason: New leader 60f39687013e4b65afd587e9d08f2d95. Latest consensus state: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.541625 9833 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:52:58.543524 9801 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.543737 9801 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:52:58.543808 9831 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.543989 9831 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:52:58.547232 9833 tablet_replica.cc:440] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60f39687013e4b65afd587e9d08f2d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d99bb94e30148eebc368262a099475c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46379 } } peers { permanent_uuid: "774fad6247404200a05df7e1699c3e7b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33601 } } peers { permanent_uuid: "60f39687013e4b65afd587e9d08f2d95" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 41963 } } }
I20250112 20:52:58.547466 9833 tablet_replica.cc:443] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: This TxnStatusTablet replica's current role is: FOLLOWER
I20250112 20:52:59.539525 9843 tablet_replica.cc:331] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c: stopping tablet replica
I20250112 20:52:59.539741 9843 raft_consensus.cc:2238] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:59.540110 9843 raft_consensus.cc:2267] T 9ee36acc0f7c473db698156a7803e1d2 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:59.540871 9844 tablet_replica.cc:331] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c: stopping tablet replica
I20250112 20:52:59.540993 9844 raft_consensus.cc:2238] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:52:59.541114 9844 raft_consensus.cc:2267] T e0a3b86f728844d09f49148c15dc8500 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:59.545794 9845 tablet_replica.cc:331] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b: stopping tablet replica
I20250112 20:52:59.545996 9845 raft_consensus.cc:2238] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:52:59.546095 9846 tablet_replica.cc:331] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b: stopping tablet replica
I20250112 20:52:59.546156 9847 tablet_replica.cc:331] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95: stopping tablet replica
I20250112 20:52:59.546422 9845 raft_consensus.cc:2267] T e0a3b86f728844d09f49148c15dc8500 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:59.546377 9846 raft_consensus.cc:2238] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:52:59.546391 9848 tablet_replica.cc:331] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95: stopping tablet replica
I20250112 20:52:59.546377 9847 raft_consensus.cc:2238] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:52:59.546701 9848 raft_consensus.cc:2238] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:52:59.546725 9846 raft_consensus.cc:2267] T 9ee36acc0f7c473db698156a7803e1d2 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:59.546770 9847 raft_consensus.cc:2267] T 9ee36acc0f7c473db698156a7803e1d2 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:52:59.546878 9848 raft_consensus.cc:2267] T e0a3b86f728844d09f49148c15dc8500 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:53:00.567044 9852 tablet_replica.cc:1307] Illegal state: The tablet is not in a running state: SHUTDOWN: cancelling pending write operations
W20250112 20:53:00.567207 9852 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:41432: Illegal state: The tablet is not in a running state: SHUTDOWN
I20250112 20:53:00.567521 9852 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41432 (ReqId={client: 3b4fe1420564424e8d353ca32d1f2120, seq_no=0, attempt_no=0}) took 2014 ms. Trace:
I20250112 20:53:00.567620 9852 rpcz_store.cc:276] 0112 20:52:58.553367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0112 20:52:58.553426 (+ 59us) service_pool.cc:225] Handling call
0112 20:53:00.567483 (+2014057us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":285,"threads_started":1,"txn-participant-registration.queue_time_us":384}
W20250112 20:53:08.571554 9768 batcher.cc:441] Timed out: Failed to write batch of 1 ops to tablet e0a3b86f728844d09f49148c15dc8500 after 123 attempt(s): Failed to write to server: (no server available): Write(tablet: e0a3b86f728844d09f49148c15dc8500, num_ops: 1, num_attempts: 123) passed its deadline: Illegal state: The tablet is not in a running state: SHUTDOWN
I20250112 20:53:08.582527 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:53:08.608588 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:53:08.608863 416 tablet_replica.cc:331] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c: stopping tablet replica
I20250112 20:53:08.609026 416 raft_consensus.cc:2238] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:53:08.609228 416 raft_consensus.cc:2267] T 7046158d21c84cfe851c0fdf53de71d5 P 2d99bb94e30148eebc368262a099475c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:08.623832 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:53:08.628254 416 tablet_server.cc:178] TabletServer@127.0.104.2:0 shutting down...
I20250112 20:53:08.636221 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:53:08.636447 416 tablet_replica.cc:331] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b: stopping tablet replica
I20250112 20:53:08.636624 416 raft_consensus.cc:2238] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250112 20:53:08.636812 416 raft_consensus.cc:2267] T 7046158d21c84cfe851c0fdf53de71d5 P 774fad6247404200a05df7e1699c3e7b [term 1 FOLLOWER]: Raft consensus is shut down!
W20250112 20:53:08.648317 9726 meta_cache.cc:302] tablet e0a3b86f728844d09f49148c15dc8500: replica 774fad6247404200a05df7e1699c3e7b (127.0.104.2:33601) has failed: Network error: Client connection negotiation failed: client connection to 127.0.104.2:33601: connect: Connection refused (error 111)
I20250112 20:53:08.650883 416 tablet_server.cc:195] TabletServer@127.0.104.2:0 shutdown complete.
I20250112 20:53:08.655112 416 tablet_server.cc:178] TabletServer@127.0.104.3:0 shutting down...
I20250112 20:53:08.663578 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:53:08.663808 416 tablet_replica.cc:331] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95: stopping tablet replica
I20250112 20:53:08.663982 416 raft_consensus.cc:2238] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:08.664291 416 raft_consensus.cc:2267] T 7046158d21c84cfe851c0fdf53de71d5 P 60f39687013e4b65afd587e9d08f2d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:08.715101 416 txn_status_manager.cc:765] Waiting for 1 task(s) to stop
I20250112 20:53:13.702082 416 tablet_server.cc:195] TabletServer@127.0.104.3:0 shutdown complete.
I20250112 20:53:13.706405 416 master.cc:537] Master@127.0.104.62:35943 shutting down...
I20250112 20:53:13.716436 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:13.716737 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:13.716877 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P f38fc768f79943f493d4e782192f24e3: stopping tablet replica
I20250112 20:53:13.730746 416 master.cc:559] Master@127.0.104.62:35943 shutdown complete.
[ OK ] TxnOpDispatcherITest.TxnWriteWhenReplicaIsShutdown (16726 ms)
[ RUN ] TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag
I20250112 20:53:13.743448 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:46409
I20250112 20:53:13.743968 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:53:13.746639 9877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:13.746712 9876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:13.746892 416 server_base.cc:1034] running on GCE node
W20250112 20:53:13.746887 9879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:13.754519 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:13.754601 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:13.754626 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715193754622 us; error 0 us; skew 500 ppm
I20250112 20:53:13.754778 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:13.756100 416 webserver.cc:458] Webserver started at http://127.0.104.62:39205/ using document root <none> and password file <none>
I20250112 20:53:13.756320 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:13.756376 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:13.756471 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:13.757076 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "f000fef85d0c47f8b63729345ce2f33a"
format_stamp: "Formatted at 2025-01-12 20:53:13 on dist-test-slave-3gql"
I20250112 20:53:13.759342 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250112 20:53:13.761018 9884 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:13.761325 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250112 20:53:13.761426 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "f000fef85d0c47f8b63729345ce2f33a"
format_stamp: "Formatted at 2025-01-12 20:53:13 on dist-test-slave-3gql"
I20250112 20:53:13.761536 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:13.775087 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:13.775597 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:13.787143 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46409
I20250112 20:53:13.787181 9945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46409 every 8 connection(s)
I20250112 20:53:13.788374 9946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:13.792234 9946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a: Bootstrap starting.
I20250112 20:53:13.793330 9946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:13.794641 9946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a: No bootstrap required, opened a new log
I20250112 20:53:13.795214 9946 raft_consensus.cc:357] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000fef85d0c47f8b63729345ce2f33a" member_type: VOTER }
I20250112 20:53:13.795338 9946 raft_consensus.cc:383] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:13.795400 9946 raft_consensus.cc:738] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f000fef85d0c47f8b63729345ce2f33a, State: Initialized, Role: FOLLOWER
I20250112 20:53:13.795560 9946 consensus_queue.cc:260] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000fef85d0c47f8b63729345ce2f33a" member_type: VOTER }
I20250112 20:53:13.795656 9946 raft_consensus.cc:397] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:13.795701 9946 raft_consensus.cc:491] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:13.795755 9946 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:13.797557 9946 raft_consensus.cc:513] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000fef85d0c47f8b63729345ce2f33a" member_type: VOTER }
I20250112 20:53:13.797744 9946 leader_election.cc:304] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f000fef85d0c47f8b63729345ce2f33a; no voters:
I20250112 20:53:13.798125 9946 leader_election.cc:290] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:13.798219 9949 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:13.798640 9949 raft_consensus.cc:695] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 1 LEADER]: Becoming Leader. State: Replica: f000fef85d0c47f8b63729345ce2f33a, State: Running, Role: LEADER
I20250112 20:53:13.798844 9949 consensus_queue.cc:237] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000fef85d0c47f8b63729345ce2f33a" member_type: VOTER }
I20250112 20:53:13.798910 9946 sys_catalog.cc:564] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:53:13.799592 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f000fef85d0c47f8b63729345ce2f33a. Latest consensus state: current_term: 1 leader_uuid: "f000fef85d0c47f8b63729345ce2f33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000fef85d0c47f8b63729345ce2f33a" member_type: VOTER } }
I20250112 20:53:13.799645 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f000fef85d0c47f8b63729345ce2f33a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f000fef85d0c47f8b63729345ce2f33a" member_type: VOTER } }
I20250112 20:53:13.799813 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:13.799863 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:13.803700 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
W20250112 20:53:13.804704 9965 catalog_manager.cc:1559] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250112 20:53:13.804806 9965 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250112 20:53:13.804920 9954 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:53:13.805881 9954 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:53:13.808605 9954 catalog_manager.cc:1348] Generated new cluster ID: 656d4012ea384d5f9fe440390bc6322f
I20250112 20:53:13.808686 9954 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:53:13.819643 9954 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:53:13.820294 9954 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:53:13.833720 9954 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a: Generated new TSK 0
I20250112 20:53:13.833926 9954 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:53:13.836215 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:53:13.838610 9974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:13.838608 9973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:13.839021 9976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:13.839378 416 server_base.cc:1034] running on GCE node
I20250112 20:53:13.839655 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:13.839711 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:13.839744 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715193839741 us; error 0 us; skew 500 ppm
I20250112 20:53:13.839892 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:13.841413 416 webserver.cc:458] Webserver started at http://127.0.104.1:36863/ using document root <none> and password file <none>
I20250112 20:53:13.841660 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:13.841768 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:13.841905 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:13.842654 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "f862dfcccab44a83986e48ee571eff71"
format_stamp: "Formatted at 2025-01-12 20:53:13 on dist-test-slave-3gql"
I20250112 20:53:13.843928 9901 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56846:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:53:13.845355 9946 master.cc:440] Invalid argument: unable to initialize TxnManager: Error creating table kudu_system.kudu_transactions on the master: not enough live tablet servers to create a table with the requested replication factor 1; 0 tablet servers are alive: unable to init TxnManager, will retry in 1.000s
I20250112 20:53:13.845649 416 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250112 20:53:13.847018 9981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:13.847308 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:53:13.847402 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "f862dfcccab44a83986e48ee571eff71"
format_stamp: "Formatted at 2025-01-12 20:53:13 on dist-test-slave-3gql"
I20250112 20:53:13.847517 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:13.860188 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:13.860764 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:13.862452 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:53:13.862567 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:13.862682 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:53:13.862732 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:13.878819 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:44817
I20250112 20:53:13.878839 10050 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:44817 every 8 connection(s)
I20250112 20:53:13.880431 10051 heartbeater.cc:346] Connected to a master server at 127.0.104.62:46409
I20250112 20:53:13.880527 10051 heartbeater.cc:463] Registering TS with master...
I20250112 20:53:13.880822 10051 heartbeater.cc:510] Master 127.0.104.62:46409 requested a full tablet report, sending...
I20250112 20:53:13.881477 9901 ts_manager.cc:194] Registered new tserver with Master: f862dfcccab44a83986e48ee571eff71 (127.0.104.1:44817)
I20250112 20:53:13.882135 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002284431s
I20250112 20:53:13.882330 9901 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56848
I20250112 20:53:13.891366 9901 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56864:
name: "txn_op_dispatcher_test"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 2
seed: 0
}
}
I20250112 20:53:13.903128 10011 tablet_service.cc:1467] Processing CreateTablet for tablet 2d4f106485be4041ba815965a660ff4d (DEFAULT_TABLE table=txn_op_dispatcher_test [id=882f4b27f28b45248764868ecb51b442]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:13.903220 10010 tablet_service.cc:1467] Processing CreateTablet for tablet 9f0d60b4366f4e32b2aec94d171cd5c2 (DEFAULT_TABLE table=txn_op_dispatcher_test [id=882f4b27f28b45248764868ecb51b442]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:13.903635 10011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4f106485be4041ba815965a660ff4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:13.903825 10010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f0d60b4366f4e32b2aec94d171cd5c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:13.908272 10061 tablet_bootstrap.cc:492] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71: Bootstrap starting.
I20250112 20:53:13.909451 10061 tablet_bootstrap.cc:654] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:13.910995 10061 tablet_bootstrap.cc:492] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71: No bootstrap required, opened a new log
I20250112 20:53:13.911111 10061 ts_tablet_manager.cc:1397] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:13.911616 10061 raft_consensus.cc:357] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.911732 10061 raft_consensus.cc:383] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:13.911778 10061 raft_consensus.cc:738] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f862dfcccab44a83986e48ee571eff71, State: Initialized, Role: FOLLOWER
I20250112 20:53:13.911962 10061 consensus_queue.cc:260] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.912088 10061 raft_consensus.cc:397] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:13.912124 10061 raft_consensus.cc:491] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:13.912204 10061 raft_consensus.cc:3054] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:13.913825 10061 raft_consensus.cc:513] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.914027 10061 leader_election.cc:304] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f862dfcccab44a83986e48ee571eff71; no voters:
I20250112 20:53:13.914407 10061 leader_election.cc:290] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:13.914551 10063 raft_consensus.cc:2798] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:13.915032 10051 heartbeater.cc:502] Master 127.0.104.62:46409 was elected leader, sending a full tablet report...
I20250112 20:53:13.915077 10061 ts_tablet_manager.cc:1428] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:13.914991 10063 raft_consensus.cc:695] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 1 LEADER]: Becoming Leader. State: Replica: f862dfcccab44a83986e48ee571eff71, State: Running, Role: LEADER
I20250112 20:53:13.915503 10061 tablet_bootstrap.cc:492] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71: Bootstrap starting.
I20250112 20:53:13.915511 10063 consensus_queue.cc:237] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.916764 10061 tablet_bootstrap.cc:654] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:13.917742 9901 catalog_manager.cc:5526] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 reported cstate change: term changed from 0 to 1, leader changed from <none> to f862dfcccab44a83986e48ee571eff71 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f862dfcccab44a83986e48ee571eff71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:13.918174 10061 tablet_bootstrap.cc:492] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71: No bootstrap required, opened a new log
I20250112 20:53:13.918262 10061 ts_tablet_manager.cc:1397] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:13.918733 10061 raft_consensus.cc:357] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.918839 10061 raft_consensus.cc:383] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:13.918921 10061 raft_consensus.cc:738] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f862dfcccab44a83986e48ee571eff71, State: Initialized, Role: FOLLOWER
I20250112 20:53:13.919106 10061 consensus_queue.cc:260] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.919224 10061 raft_consensus.cc:397] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:13.919312 10061 raft_consensus.cc:491] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:13.919384 10061 raft_consensus.cc:3054] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:13.921869 10061 raft_consensus.cc:513] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.922082 10061 leader_election.cc:304] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f862dfcccab44a83986e48ee571eff71; no voters:
I20250112 20:53:13.922248 10061 leader_election.cc:290] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:13.922297 10063 raft_consensus.cc:2798] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:13.922441 10063 raft_consensus.cc:695] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 1 LEADER]: Becoming Leader. State: Replica: f862dfcccab44a83986e48ee571eff71, State: Running, Role: LEADER
I20250112 20:53:13.922683 10061 ts_tablet_manager.cc:1428] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:13.922668 10063 consensus_queue.cc:237] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:13.924218 9900 catalog_manager.cc:5526] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 reported cstate change: term changed from 0 to 1, leader changed from <none> to f862dfcccab44a83986e48ee571eff71 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f862dfcccab44a83986e48ee571eff71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } health_report { overall_health: HEALTHY } } }
W20250112 20:53:13.931167 10054 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:53:14.857368 9900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41576:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:53:14.864110 10011 tablet_service.cc:1467] Processing CreateTablet for tablet 9647d6b07e2e4bf4806ecbd8151368d5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7ce567ce0c84da4bbc31e4a7b90cade]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:53:14.864609 10011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9647d6b07e2e4bf4806ecbd8151368d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:14.868641 10079 tablet_bootstrap.cc:492] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: Bootstrap starting.
I20250112 20:53:14.869966 10079 tablet_bootstrap.cc:654] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:14.871328 10079 tablet_bootstrap.cc:492] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: No bootstrap required, opened a new log
I20250112 20:53:14.871446 10079 ts_tablet_manager.cc:1397] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:14.871944 10079 raft_consensus.cc:357] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:14.872077 10079 raft_consensus.cc:383] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:14.872164 10079 raft_consensus.cc:738] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f862dfcccab44a83986e48ee571eff71, State: Initialized, Role: FOLLOWER
I20250112 20:53:14.872362 10079 consensus_queue.cc:260] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:14.872500 10079 raft_consensus.cc:397] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:14.872587 10079 raft_consensus.cc:491] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:14.872663 10079 raft_consensus.cc:3054] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:14.874466 10079 raft_consensus.cc:513] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:14.874624 10079 leader_election.cc:304] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f862dfcccab44a83986e48ee571eff71; no voters:
I20250112 20:53:14.874929 10079 leader_election.cc:290] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:14.875077 10081 raft_consensus.cc:2798] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:14.875505 10081 raft_consensus.cc:695] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 1 LEADER]: Becoming Leader. State: Replica: f862dfcccab44a83986e48ee571eff71, State: Running, Role: LEADER
I20250112 20:53:14.875746 10081 consensus_queue.cc:237] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } }
I20250112 20:53:14.875823 10079 ts_tablet_manager.cc:1428] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20250112 20:53:14.876621 10082 tablet_replica.cc:440] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f862dfcccab44a83986e48ee571eff71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } } }
I20250112 20:53:14.877317 10082 tablet_replica.cc:443] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:53:14.877820 9900 catalog_manager.cc:5526] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 reported cstate change: term changed from 0 to 1, leader changed from <none> to f862dfcccab44a83986e48ee571eff71 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f862dfcccab44a83986e48ee571eff71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:14.878767 10086 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:53:14.878890 10086 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:53:14.878887 10083 tablet_replica.cc:440] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: TxnStatusTablet state changed. Reason: New leader f862dfcccab44a83986e48ee571eff71. Latest consensus state: current_term: 1 leader_uuid: "f862dfcccab44a83986e48ee571eff71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f862dfcccab44a83986e48ee571eff71" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44817 } } }
I20250112 20:53:14.879078 10083 tablet_replica.cc:443] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: This TxnStatusTablet replica's current role is: LEADER
W20250112 20:53:15.902078 10090 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60648: Not found: txn 0 not found on tablet 2d4f106485be4041ba815965a660ff4d
W20250112 20:53:15.902623 10055 batcher.cc:441] Not found: Failed to write batch of 1 ops to tablet 2d4f106485be4041ba815965a660ff4d after 1 attempt(s): Failed to write to server: f862dfcccab44a83986e48ee571eff71 (127.0.104.1:44817): txn 0 not found on tablet 2d4f106485be4041ba815965a660ff4d
I20250112 20:53:15.918380 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:53:15.926297 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:53:15.926558 416 tablet_replica.cc:331] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71: stopping tablet replica
I20250112 20:53:15.926728 416 raft_consensus.cc:2238] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:15.926929 416 raft_consensus.cc:2267] T 9647d6b07e2e4bf4806ecbd8151368d5 P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:15.927629 416 tablet_replica.cc:331] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71: stopping tablet replica
I20250112 20:53:15.927769 416 raft_consensus.cc:2238] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:15.927911 416 raft_consensus.cc:2267] T 2d4f106485be4041ba815965a660ff4d P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:15.928324 416 tablet_replica.cc:331] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71: stopping tablet replica
I20250112 20:53:15.928455 416 raft_consensus.cc:2238] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:15.928611 416 raft_consensus.cc:2267] T 9f0d60b4366f4e32b2aec94d171cd5c2 P f862dfcccab44a83986e48ee571eff71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:15.932338 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:53:15.935949 416 master.cc:537] Master@127.0.104.62:46409 shutting down...
I20250112 20:53:15.943770 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:15.943985 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:15.944087 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P f000fef85d0c47f8b63729345ce2f33a: stopping tablet replica
I20250112 20:53:15.957326 416 master.cc:559] Master@127.0.104.62:46409 shutdown complete.
[ OK ] TxnOpDispatcherITest.TxnWriteOpHandlingEnabledFlag (2224 ms)
[ DISABLED ] TxnOpDispatcherITest.DISABLED_TxnMultipleSingleRowsWithServerRestart
[ RUN ] TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload
I20250112 20:53:15.967677 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:37037
I20250112 20:53:15.968125 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:53:15.970230 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:15.970364 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:15.971484 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:15.971524 416 server_base.cc:1034] running on GCE node
I20250112 20:53:15.971758 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:15.971810 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:15.971845 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715195971841 us; error 0 us; skew 500 ppm
I20250112 20:53:15.972007 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:15.973335 416 webserver.cc:458] Webserver started at http://127.0.104.62:39261/ using document root <none> and password file <none>
I20250112 20:53:15.973544 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:15.973606 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:15.973734 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:15.974434 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "96f462af80004fd69c1175eeee5f462a"
format_stamp: "Formatted at 2025-01-12 20:53:15 on dist-test-slave-3gql"
I20250112 20:53:15.977005 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250112 20:53:15.978446 10105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:15.978837 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250112 20:53:15.978919 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "96f462af80004fd69c1175eeee5f462a"
format_stamp: "Formatted at 2025-01-12 20:53:15 on dist-test-slave-3gql"
I20250112 20:53:15.979007 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:15.990772 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:15.991173 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:16.002892 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37037
I20250112 20:53:16.002907 10166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37037 every 8 connection(s)
I20250112 20:53:16.004024 10167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:16.007684 10167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a: Bootstrap starting.
I20250112 20:53:16.008808 10167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:16.010141 10167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a: No bootstrap required, opened a new log
I20250112 20:53:16.010689 10167 raft_consensus.cc:357] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f462af80004fd69c1175eeee5f462a" member_type: VOTER }
I20250112 20:53:16.010818 10167 raft_consensus.cc:383] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:16.010870 10167 raft_consensus.cc:738] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96f462af80004fd69c1175eeee5f462a, State: Initialized, Role: FOLLOWER
I20250112 20:53:16.011067 10167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f462af80004fd69c1175eeee5f462a" member_type: VOTER }
I20250112 20:53:16.011168 10167 raft_consensus.cc:397] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:16.011233 10167 raft_consensus.cc:491] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:16.011312 10167 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:16.012841 10167 raft_consensus.cc:513] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f462af80004fd69c1175eeee5f462a" member_type: VOTER }
I20250112 20:53:16.013005 10167 leader_election.cc:304] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96f462af80004fd69c1175eeee5f462a; no voters:
I20250112 20:53:16.013365 10167 leader_election.cc:290] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:16.013479 10170 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:16.013844 10170 raft_consensus.cc:695] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 1 LEADER]: Becoming Leader. State: Replica: 96f462af80004fd69c1175eeee5f462a, State: Running, Role: LEADER
I20250112 20:53:16.014055 10170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f462af80004fd69c1175eeee5f462a" member_type: VOTER }
I20250112 20:53:16.014117 10167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:53:16.014672 10171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96f462af80004fd69c1175eeee5f462a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f462af80004fd69c1175eeee5f462a" member_type: VOTER } }
I20250112 20:53:16.014853 10171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:16.014715 10172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96f462af80004fd69c1175eeee5f462a. Latest consensus state: current_term: 1 leader_uuid: "96f462af80004fd69c1175eeee5f462a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96f462af80004fd69c1175eeee5f462a" member_type: VOTER } }
I20250112 20:53:16.014990 10172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:16.015878 10175 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:53:16.017100 10175 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:53:16.018661 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:53:16.019748 10175 catalog_manager.cc:1348] Generated new cluster ID: c806e89496414985ae8194d1c00eebbc
I20250112 20:53:16.019824 10175 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:53:16.031402 10175 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:53:16.032106 10175 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:53:16.044078 10175 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a: Generated new TSK 0
I20250112 20:53:16.044307 10175 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:53:16.051227 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250112 20:53:16.053025 10122 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42944:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20250112 20:53:16.053874 10195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:16.053874 10194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:16.055017 10197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:16.055426 416 server_base.cc:1034] running on GCE node
I20250112 20:53:16.055682 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:16.055737 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:16.055771 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715196055767 us; error 0 us; skew 500 ppm
I20250112 20:53:16.055917 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:16.057053 416 webserver.cc:458] Webserver started at http://127.0.104.1:38231/ using document root <none> and password file <none>
I20250112 20:53:16.057260 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:16.057327 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:16.057452 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:16.058041 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "18c702090b0c434bafc2e544215788ed"
format_stamp: "Formatted at 2025-01-12 20:53:16 on dist-test-slave-3gql"
I20250112 20:53:16.060362 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:53:16.061820 10202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:16.062103 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:53:16.062192 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "18c702090b0c434bafc2e544215788ed"
format_stamp: "Formatted at 2025-01-12 20:53:16 on dist-test-slave-3gql"
I20250112 20:53:16.062300 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:16.074618 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:16.075093 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:16.076750 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:53:16.076881 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:16.076979 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:53:16.077035 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:16.093176 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33453
I20250112 20:53:16.093204 10271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33453 every 8 connection(s)
I20250112 20:53:16.094657 10272 heartbeater.cc:346] Connected to a master server at 127.0.104.62:37037
I20250112 20:53:16.094769 10272 heartbeater.cc:463] Registering TS with master...
I20250112 20:53:16.095062 10272 heartbeater.cc:510] Master 127.0.104.62:37037 requested a full tablet report, sending...
I20250112 20:53:16.095758 10122 ts_manager.cc:194] Registered new tserver with Master: 18c702090b0c434bafc2e544215788ed (127.0.104.1:33453)
I20250112 20:53:16.096418 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002279029s
I20250112 20:53:16.096844 10122 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42948
I20250112 20:53:16.097517 416 test_util.cc:274] Using random seed: -2074680344
I20250112 20:53:16.107795 10122 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42976:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250112 20:53:16.119879 10234 tablet_service.cc:1467] Processing CreateTablet for tablet e1f5351ce0ea4cc98af39c25c11ac4db (DEFAULT_TABLE table=test-workload [id=d9fc3f595c4c42a4965a491a79158c32]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:16.119937 10232 tablet_service.cc:1467] Processing CreateTablet for tablet b125dee29e7a467d9c783133347ba2b2 (DEFAULT_TABLE table=test-workload [id=d9fc3f595c4c42a4965a491a79158c32]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:16.119903 10230 tablet_service.cc:1467] Processing CreateTablet for tablet 924d6edbd3d640b4ba3c0ccc59e6375c (DEFAULT_TABLE table=test-workload [id=d9fc3f595c4c42a4965a491a79158c32]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:16.120414 10234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1f5351ce0ea4cc98af39c25c11ac4db. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:16.120625 10230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 924d6edbd3d640b4ba3c0ccc59e6375c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:16.120826 10232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125dee29e7a467d9c783133347ba2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:16.124941 10282 tablet_bootstrap.cc:492] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed: Bootstrap starting.
I20250112 20:53:16.126192 10282 tablet_bootstrap.cc:654] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:16.127645 10282 tablet_bootstrap.cc:492] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed: No bootstrap required, opened a new log
I20250112 20:53:16.127744 10282 ts_tablet_manager.cc:1397] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:16.128248 10282 raft_consensus.cc:357] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.128372 10282 raft_consensus.cc:383] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:16.128404 10282 raft_consensus.cc:738] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Initialized, Role: FOLLOWER
I20250112 20:53:16.128613 10282 consensus_queue.cc:260] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.128728 10282 raft_consensus.cc:397] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:16.128780 10282 raft_consensus.cc:491] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:16.128847 10282 raft_consensus.cc:3054] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:16.130352 10282 raft_consensus.cc:513] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.130525 10282 leader_election.cc:304] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c702090b0c434bafc2e544215788ed; no voters:
I20250112 20:53:16.130867 10282 leader_election.cc:290] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:16.130985 10284 raft_consensus.cc:2798] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:16.131448 10272 heartbeater.cc:502] Master 127.0.104.62:37037 was elected leader, sending a full tablet report...
I20250112 20:53:16.131494 10282 ts_tablet_manager.cc:1428] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:16.131705 10282 tablet_bootstrap.cc:492] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed: Bootstrap starting.
I20250112 20:53:16.132771 10284 raft_consensus.cc:695] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Becoming Leader. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Running, Role: LEADER
I20250112 20:53:16.132932 10282 tablet_bootstrap.cc:654] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:16.133046 10284 consensus_queue.cc:237] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.134601 10282 tablet_bootstrap.cc:492] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed: No bootstrap required, opened a new log
I20250112 20:53:16.134768 10282 ts_tablet_manager.cc:1397] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:53:16.135603 10282 raft_consensus.cc:357] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.135500 10122 catalog_manager.cc:5526] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c702090b0c434bafc2e544215788ed (127.0.104.1). New cstate: current_term: 1 leader_uuid: "18c702090b0c434bafc2e544215788ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:16.135731 10282 raft_consensus.cc:383] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:16.135780 10282 raft_consensus.cc:738] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Initialized, Role: FOLLOWER
I20250112 20:53:16.135951 10282 consensus_queue.cc:260] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.136101 10282 raft_consensus.cc:397] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:16.136160 10282 raft_consensus.cc:491] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:16.136233 10282 raft_consensus.cc:3054] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:16.138643 10282 raft_consensus.cc:513] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.138809 10282 leader_election.cc:304] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c702090b0c434bafc2e544215788ed; no voters:
I20250112 20:53:16.138963 10282 leader_election.cc:290] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:16.139027 10284 raft_consensus.cc:2798] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:16.139271 10284 raft_consensus.cc:695] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Becoming Leader. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Running, Role: LEADER
I20250112 20:53:16.139456 10282 ts_tablet_manager.cc:1428] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20250112 20:53:16.139530 10284 consensus_queue.cc:237] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.139720 10282 tablet_bootstrap.cc:492] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed: Bootstrap starting.
I20250112 20:53:16.141237 10282 tablet_bootstrap.cc:654] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:16.142146 10122 catalog_manager.cc:5526] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c702090b0c434bafc2e544215788ed (127.0.104.1). New cstate: current_term: 1 leader_uuid: "18c702090b0c434bafc2e544215788ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:16.142908 10282 tablet_bootstrap.cc:492] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed: No bootstrap required, opened a new log
I20250112 20:53:16.143023 10282 ts_tablet_manager.cc:1397] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:53:16.143604 10282 raft_consensus.cc:357] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.143744 10282 raft_consensus.cc:383] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:16.143862 10282 raft_consensus.cc:738] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Initialized, Role: FOLLOWER
I20250112 20:53:16.144044 10282 consensus_queue.cc:260] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.144158 10282 raft_consensus.cc:397] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:16.144235 10282 raft_consensus.cc:491] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:16.144338 10282 raft_consensus.cc:3054] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:16.146358 10282 raft_consensus.cc:513] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.146523 10282 leader_election.cc:304] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c702090b0c434bafc2e544215788ed; no voters:
I20250112 20:53:16.146668 10282 leader_election.cc:290] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:16.146728 10284 raft_consensus.cc:2798] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:16.146893 10284 raft_consensus.cc:695] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Becoming Leader. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Running, Role: LEADER
I20250112 20:53:16.147027 10282 ts_tablet_manager.cc:1428] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250112 20:53:16.147079 10284 consensus_queue.cc:237] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:16.148691 10121 catalog_manager.cc:5526] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c702090b0c434bafc2e544215788ed (127.0.104.1). New cstate: current_term: 1 leader_uuid: "18c702090b0c434bafc2e544215788ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } health_report { overall_health: HEALTHY } } }
W20250112 20:53:16.154261 10277 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:53:17.071141 10121 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42978:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:53:17.078078 10232 tablet_service.cc:1467] Processing CreateTablet for tablet f4314050b314476a991c23b20c566b21 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a9a55f8fb4b84a1f84e499b5282f0596]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:53:17.078505 10232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4314050b314476a991c23b20c566b21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:17.082712 10302 tablet_bootstrap.cc:492] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: Bootstrap starting.
I20250112 20:53:17.084383 10302 tablet_bootstrap.cc:654] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:17.086014 10302 tablet_bootstrap.cc:492] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: No bootstrap required, opened a new log
I20250112 20:53:17.086145 10302 ts_tablet_manager.cc:1397] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:53:17.086726 10302 raft_consensus.cc:357] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:17.086926 10302 raft_consensus.cc:383] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:17.087015 10302 raft_consensus.cc:738] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Initialized, Role: FOLLOWER
I20250112 20:53:17.087241 10302 consensus_queue.cc:260] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:17.087378 10302 raft_consensus.cc:397] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:17.087433 10302 raft_consensus.cc:491] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:17.087502 10302 raft_consensus.cc:3054] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:17.089179 10302 raft_consensus.cc:513] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:17.089360 10302 leader_election.cc:304] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18c702090b0c434bafc2e544215788ed; no voters:
I20250112 20:53:17.089706 10302 leader_election.cc:290] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:17.089830 10304 raft_consensus.cc:2798] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:17.090325 10304 raft_consensus.cc:695] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Becoming Leader. State: Replica: 18c702090b0c434bafc2e544215788ed, State: Running, Role: LEADER
I20250112 20:53:17.090385 10302 ts_tablet_manager.cc:1428] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:17.090826 10304 consensus_queue.cc:237] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } }
I20250112 20:53:17.091717 10306 tablet_replica.cc:440] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: TxnStatusTablet state changed. Reason: New leader 18c702090b0c434bafc2e544215788ed. Latest consensus state: current_term: 1 leader_uuid: "18c702090b0c434bafc2e544215788ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } } }
I20250112 20:53:17.091813 10305 tablet_replica.cc:440] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18c702090b0c434bafc2e544215788ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } } }
I20250112 20:53:17.091921 10306 tablet_replica.cc:443] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:53:17.091986 10305 tablet_replica.cc:443] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:53:17.093016 10121 catalog_manager.cc:5526] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 18c702090b0c434bafc2e544215788ed (127.0.104.1). New cstate: current_term: 1 leader_uuid: "18c702090b0c434bafc2e544215788ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18c702090b0c434bafc2e544215788ed" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33453 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:17.101116 10308 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:53:17.101282 10308 txn_status_manager.cc:930] Loading transaction status metadata into memory...
W20250112 20:53:17.368178 10328 tablet_replica.cc:1307] Aborted: operation has been aborted: cancelling pending write operations
I20250112 20:53:17.383502 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:53:17.398806 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:53:17.399726 416 tablet_replica.cc:331] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed: stopping tablet replica
I20250112 20:53:17.399933 416 raft_consensus.cc:2238] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:17.400128 416 raft_consensus.cc:2267] T f4314050b314476a991c23b20c566b21 P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:17.401170 416 tablet_replica.cc:331] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed: stopping tablet replica
I20250112 20:53:17.401319 416 raft_consensus.cc:2238] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:17.401463 416 raft_consensus.cc:2267] T b125dee29e7a467d9c783133347ba2b2 P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:17.402042 416 tablet_replica.cc:331] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed: stopping tablet replica
I20250112 20:53:17.402164 416 raft_consensus.cc:2238] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:17.402292 416 raft_consensus.cc:2267] T e1f5351ce0ea4cc98af39c25c11ac4db P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:17.402863 416 tablet_replica.cc:331] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed: stopping tablet replica
I20250112 20:53:17.402987 416 raft_consensus.cc:2238] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:17.403117 416 raft_consensus.cc:2267] T 924d6edbd3d640b4ba3c0ccc59e6375c P 18c702090b0c434bafc2e544215788ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:17.409220 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:53:17.415086 416 master.cc:537] Master@127.0.104.62:37037 shutting down...
I20250112 20:53:17.424258 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:17.424468 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:17.424660 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 96f462af80004fd69c1175eeee5f462a: stopping tablet replica
I20250112 20:53:17.438794 416 master.cc:559] Master@127.0.104.62:37037 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestBeginAbortTransactionalTestWorkload (1484 ms)
[ RUN ] TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload
I20250112 20:53:17.452976 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:37577
I20250112 20:53:17.453609 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:53:17.456061 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:17.456514 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:17.456830 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:17.457526 416 server_base.cc:1034] running on GCE node
I20250112 20:53:17.457789 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:17.457836 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:17.457859 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715197457855 us; error 0 us; skew 500 ppm
I20250112 20:53:17.457995 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:17.459373 416 webserver.cc:458] Webserver started at http://127.0.104.62:36963/ using document root <none> and password file <none>
I20250112 20:53:17.459590 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:17.459661 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:17.459779 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:17.460381 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "94098b4d282f4d2eba12db131bd98eb0"
format_stamp: "Formatted at 2025-01-12 20:53:17 on dist-test-slave-3gql"
I20250112 20:53:17.462878 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20250112 20:53:17.464440 10342 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:17.464830 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250112 20:53:17.464923 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "94098b4d282f4d2eba12db131bd98eb0"
format_stamp: "Formatted at 2025-01-12 20:53:17 on dist-test-slave-3gql"
I20250112 20:53:17.465050 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:17.494716 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:17.495276 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:17.511791 10403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37577 every 8 connection(s)
I20250112 20:53:17.511785 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37577
I20250112 20:53:17.513236 10404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:17.517194 10404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0: Bootstrap starting.
I20250112 20:53:17.518393 10404 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:17.519850 10404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0: No bootstrap required, opened a new log
I20250112 20:53:17.520449 10404 raft_consensus.cc:357] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94098b4d282f4d2eba12db131bd98eb0" member_type: VOTER }
I20250112 20:53:17.520620 10404 raft_consensus.cc:383] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:17.520669 10404 raft_consensus.cc:738] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94098b4d282f4d2eba12db131bd98eb0, State: Initialized, Role: FOLLOWER
I20250112 20:53:17.520864 10404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94098b4d282f4d2eba12db131bd98eb0" member_type: VOTER }
I20250112 20:53:17.520977 10404 raft_consensus.cc:397] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:17.521032 10404 raft_consensus.cc:491] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:17.521098 10404 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:17.522711 10404 raft_consensus.cc:513] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94098b4d282f4d2eba12db131bd98eb0" member_type: VOTER }
I20250112 20:53:17.522872 10404 leader_election.cc:304] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94098b4d282f4d2eba12db131bd98eb0; no voters:
I20250112 20:53:17.523272 10404 leader_election.cc:290] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:17.523419 10407 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:17.524013 10404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:53:17.524735 10407 raft_consensus.cc:695] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 94098b4d282f4d2eba12db131bd98eb0, State: Running, Role: LEADER
I20250112 20:53:17.525024 10407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94098b4d282f4d2eba12db131bd98eb0" member_type: VOTER }
I20250112 20:53:17.526340 10408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94098b4d282f4d2eba12db131bd98eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94098b4d282f4d2eba12db131bd98eb0" member_type: VOTER } }
I20250112 20:53:17.526546 10408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:17.526335 10409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94098b4d282f4d2eba12db131bd98eb0. Latest consensus state: current_term: 1 leader_uuid: "94098b4d282f4d2eba12db131bd98eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94098b4d282f4d2eba12db131bd98eb0" member_type: VOTER } }
I20250112 20:53:17.526971 10409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:17.527146 10415 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:53:17.529178 10415 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:53:17.535256 10415 catalog_manager.cc:1348] Generated new cluster ID: 6fb9b3df8fd2491fa94ba7183818d72d
I20250112 20:53:17.535353 10415 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:53:17.539287 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:53:17.544639 10415 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:53:17.545647 10415 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:53:17.556113 10415 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0: Generated new TSK 0
I20250112 20:53:17.556370 10415 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:53:17.561990 10359 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49106:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:53:17.571988 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:53:17.574683 10431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:17.575142 10432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:17.576419 10434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:17.576666 416 server_base.cc:1034] running on GCE node
I20250112 20:53:17.576910 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:17.576961 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:17.576983 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715197576980 us; error 0 us; skew 500 ppm
I20250112 20:53:17.577133 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:17.578564 416 webserver.cc:458] Webserver started at http://127.0.104.1:38737/ using document root <none> and password file <none>
I20250112 20:53:17.578778 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:17.578842 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:17.578985 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:17.579674 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "6b4be2b884f24c58b02a24aa803701ab"
format_stamp: "Formatted at 2025-01-12 20:53:17 on dist-test-slave-3gql"
I20250112 20:53:17.588503 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20250112 20:53:17.590205 10439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:17.590579 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250112 20:53:17.590669 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "6b4be2b884f24c58b02a24aa803701ab"
format_stamp: "Formatted at 2025-01-12 20:53:17 on dist-test-slave-3gql"
I20250112 20:53:17.590795 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:17.598567 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:17.599011 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:17.601125 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:53:17.601254 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:17.601368 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:53:17.601425 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:17.621133 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:46161
I20250112 20:53:17.621181 10508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:46161 every 8 connection(s)
I20250112 20:53:17.622625 10509 heartbeater.cc:346] Connected to a master server at 127.0.104.62:37577
I20250112 20:53:17.622730 10509 heartbeater.cc:463] Registering TS with master...
I20250112 20:53:17.623011 10509 heartbeater.cc:510] Master 127.0.104.62:37577 requested a full tablet report, sending...
I20250112 20:53:17.623801 10359 ts_manager.cc:194] Registered new tserver with Master: 6b4be2b884f24c58b02a24aa803701ab (127.0.104.1:46161)
I20250112 20:53:17.624339 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.00226289s
I20250112 20:53:17.625032 10359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49118
I20250112 20:53:17.625582 416 test_util.cc:274] Using random seed: -2073152278
I20250112 20:53:17.636830 10359 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49134:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250112 20:53:17.658506 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 7894817c70d24bc7a5a2773dd157192f (DEFAULT_TABLE table=test-workload [id=8925db6c45be45f2967e6aa809d0e97c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:17.658541 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 6adcfc3de77443548a2809a046c3f235 (DEFAULT_TABLE table=test-workload [id=8925db6c45be45f2967e6aa809d0e97c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:17.659013 10471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7894817c70d24bc7a5a2773dd157192f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:17.658510 10467 tablet_service.cc:1467] Processing CreateTablet for tablet 4797bf6a96f9439582e05124cec3d268 (DEFAULT_TABLE table=test-workload [id=8925db6c45be45f2967e6aa809d0e97c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:17.659174 10469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6adcfc3de77443548a2809a046c3f235. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:17.659370 10467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4797bf6a96f9439582e05124cec3d268. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:17.665632 10519 tablet_bootstrap.cc:492] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab: Bootstrap starting.
I20250112 20:53:17.667270 10519 tablet_bootstrap.cc:654] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:17.668910 10519 tablet_bootstrap.cc:492] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab: No bootstrap required, opened a new log
I20250112 20:53:17.669042 10519 ts_tablet_manager.cc:1397] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:53:17.669613 10519 raft_consensus.cc:357] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.669773 10519 raft_consensus.cc:383] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:17.669845 10519 raft_consensus.cc:738] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Initialized, Role: FOLLOWER
I20250112 20:53:17.670059 10519 consensus_queue.cc:260] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.670208 10519 raft_consensus.cc:397] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:17.670280 10519 raft_consensus.cc:491] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:17.670372 10519 raft_consensus.cc:3054] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:17.672363 10519 raft_consensus.cc:513] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.672610 10519 leader_election.cc:304] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4be2b884f24c58b02a24aa803701ab; no voters:
I20250112 20:53:17.673043 10519 leader_election.cc:290] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:17.673156 10521 raft_consensus.cc:2798] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:17.673760 10509 heartbeater.cc:502] Master 127.0.104.62:37577 was elected leader, sending a full tablet report...
I20250112 20:53:17.674401 10519 ts_tablet_manager.cc:1428] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250112 20:53:17.674723 10519 tablet_bootstrap.cc:492] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab: Bootstrap starting.
I20250112 20:53:17.676743 10521 raft_consensus.cc:695] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Becoming Leader. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Running, Role: LEADER
I20250112 20:53:17.676934 10519 tablet_bootstrap.cc:654] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:17.677081 10521 consensus_queue.cc:237] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.679394 10519 tablet_bootstrap.cc:492] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab: No bootstrap required, opened a new log
I20250112 20:53:17.679566 10519 ts_tablet_manager.cc:1397] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250112 20:53:17.680425 10519 raft_consensus.cc:357] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.680656 10519 raft_consensus.cc:383] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:17.680771 10519 raft_consensus.cc:738] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Initialized, Role: FOLLOWER
I20250112 20:53:17.680644 10358 catalog_manager.cc:5526] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4be2b884f24c58b02a24aa803701ab (127.0.104.1). New cstate: current_term: 1 leader_uuid: "6b4be2b884f24c58b02a24aa803701ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:17.680979 10519 consensus_queue.cc:260] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.681108 10519 raft_consensus.cc:397] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:17.681161 10519 raft_consensus.cc:491] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:17.681221 10519 raft_consensus.cc:3054] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:17.683840 10519 raft_consensus.cc:513] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.684152 10519 leader_election.cc:304] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4be2b884f24c58b02a24aa803701ab; no voters:
I20250112 20:53:17.684362 10519 leader_election.cc:290] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:17.684481 10523 raft_consensus.cc:2798] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:17.684870 10519 ts_tablet_manager.cc:1428] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20250112 20:53:17.685099 10519 tablet_bootstrap.cc:492] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab: Bootstrap starting.
I20250112 20:53:17.686312 10519 tablet_bootstrap.cc:654] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:17.688071 10519 tablet_bootstrap.cc:492] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab: No bootstrap required, opened a new log
I20250112 20:53:17.688308 10519 ts_tablet_manager.cc:1397] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20250112 20:53:17.688732 10523 raft_consensus.cc:695] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Becoming Leader. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Running, Role: LEADER
I20250112 20:53:17.689034 10523 consensus_queue.cc:237] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.689365 10519 raft_consensus.cc:357] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.689653 10519 raft_consensus.cc:383] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:17.689774 10519 raft_consensus.cc:738] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Initialized, Role: FOLLOWER
I20250112 20:53:17.690084 10519 consensus_queue.cc:260] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.690352 10519 raft_consensus.cc:397] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:17.690469 10519 raft_consensus.cc:491] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:17.690610 10519 raft_consensus.cc:3054] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:17.694402 10519 raft_consensus.cc:513] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.694772 10519 leader_election.cc:304] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4be2b884f24c58b02a24aa803701ab; no voters:
I20250112 20:53:17.695056 10519 leader_election.cc:290] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:17.695149 10527 raft_consensus.cc:2798] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:17.695775 10519 ts_tablet_manager.cc:1428] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250112 20:53:17.696161 10527 raft_consensus.cc:695] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Becoming Leader. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Running, Role: LEADER
I20250112 20:53:17.696506 10527 consensus_queue.cc:237] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:17.698989 10358 catalog_manager.cc:5526] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4be2b884f24c58b02a24aa803701ab (127.0.104.1). New cstate: current_term: 1 leader_uuid: "6b4be2b884f24c58b02a24aa803701ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:17.704033 10358 catalog_manager.cc:5526] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4be2b884f24c58b02a24aa803701ab (127.0.104.1). New cstate: current_term: 1 leader_uuid: "6b4be2b884f24c58b02a24aa803701ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } health_report { overall_health: HEALTHY } } }
W20250112 20:53:17.715253 10515 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:53:18.574743 10358 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49144:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:53:18.581113 10467 tablet_service.cc:1467] Processing CreateTablet for tablet e5e2e57bb8d54fc49e9388b8facd434e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=91eecb7ae75a47dc9032502635e992fb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:53:18.581519 10467 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e2e57bb8d54fc49e9388b8facd434e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:18.585763 10541 tablet_bootstrap.cc:492] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: Bootstrap starting.
I20250112 20:53:18.587095 10541 tablet_bootstrap.cc:654] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:18.588428 10541 tablet_bootstrap.cc:492] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: No bootstrap required, opened a new log
I20250112 20:53:18.588531 10541 ts_tablet_manager.cc:1397] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:18.589052 10541 raft_consensus.cc:357] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:18.589176 10541 raft_consensus.cc:383] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:18.589229 10541 raft_consensus.cc:738] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Initialized, Role: FOLLOWER
I20250112 20:53:18.589409 10541 consensus_queue.cc:260] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:18.589519 10541 raft_consensus.cc:397] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:18.589576 10541 raft_consensus.cc:491] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:18.589640 10541 raft_consensus.cc:3054] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:18.591372 10541 raft_consensus.cc:513] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:18.591548 10541 leader_election.cc:304] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4be2b884f24c58b02a24aa803701ab; no voters:
I20250112 20:53:18.591892 10541 leader_election.cc:290] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:18.592007 10543 raft_consensus.cc:2798] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:18.592478 10543 raft_consensus.cc:695] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Becoming Leader. State: Replica: 6b4be2b884f24c58b02a24aa803701ab, State: Running, Role: LEADER
I20250112 20:53:18.592730 10543 consensus_queue.cc:237] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } }
I20250112 20:53:18.592813 10541 ts_tablet_manager.cc:1428] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20250112 20:53:18.593423 10544 tablet_replica.cc:440] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b4be2b884f24c58b02a24aa803701ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } } }
I20250112 20:53:18.593730 10544 tablet_replica.cc:443] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:53:18.593508 10545 tablet_replica.cc:440] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: TxnStatusTablet state changed. Reason: New leader 6b4be2b884f24c58b02a24aa803701ab. Latest consensus state: current_term: 1 leader_uuid: "6b4be2b884f24c58b02a24aa803701ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } } }
I20250112 20:53:18.594442 10545 tablet_replica.cc:443] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:53:18.595023 10358 catalog_manager.cc:5526] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b4be2b884f24c58b02a24aa803701ab (127.0.104.1). New cstate: current_term: 1 leader_uuid: "6b4be2b884f24c58b02a24aa803701ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4be2b884f24c58b02a24aa803701ab" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 46161 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:18.600164 10547 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:53:18.600317 10547 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:53:18.934353 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:53:18.943987 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:53:18.944727 416 tablet_replica.cc:331] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab: stopping tablet replica
I20250112 20:53:18.945048 416 raft_consensus.cc:2238] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:18.945232 416 raft_consensus.cc:2267] T e5e2e57bb8d54fc49e9388b8facd434e P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:18.946028 416 tablet_replica.cc:331] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab: stopping tablet replica
I20250112 20:53:18.946135 416 raft_consensus.cc:2238] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:18.946240 416 raft_consensus.cc:2267] T 7894817c70d24bc7a5a2773dd157192f P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:18.946851 416 tablet_replica.cc:331] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab: stopping tablet replica
I20250112 20:53:18.946942 416 raft_consensus.cc:2238] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:18.947036 416 raft_consensus.cc:2267] T 6adcfc3de77443548a2809a046c3f235 P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:18.947566 416 tablet_replica.cc:331] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab: stopping tablet replica
I20250112 20:53:18.947654 416 raft_consensus.cc:2238] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:18.947746 416 raft_consensus.cc:2267] T 4797bf6a96f9439582e05124cec3d268 P 6b4be2b884f24c58b02a24aa803701ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:18.963387 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:53:18.968014 416 master.cc:537] Master@127.0.104.62:37577 shutting down...
I20250112 20:53:18.978528 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:18.978749 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:18.978847 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P 94098b4d282f4d2eba12db131bd98eb0: stopping tablet replica
I20250112 20:53:18.992604 416 master.cc:559] Master@127.0.104.62:37577 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestBeginCommitTransactionalTestWorkload (1551 ms)
[ RUN ] TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads
I20250112 20:53:19.003374 416 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.104.62:43453
I20250112 20:53:19.003844 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:53:19.005940 10572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:19.006203 10573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:19.006626 10575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:19.007150 416 server_base.cc:1034] running on GCE node
I20250112 20:53:19.007378 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:19.007426 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:19.007450 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715199007445 us; error 0 us; skew 500 ppm
I20250112 20:53:19.007622 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:19.008903 416 webserver.cc:458] Webserver started at http://127.0.104.62:42403/ using document root <none> and password file <none>
I20250112 20:53:19.009125 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:19.009181 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:19.009326 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:19.009907 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/master-0-root/instance:
uuid: "b2ef36fc26c34975b2271f00d73148f3"
format_stamp: "Formatted at 2025-01-12 20:53:19 on dist-test-slave-3gql"
I20250112 20:53:19.012296 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250112 20:53:19.013757 10580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:19.014117 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:53:19.014266 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/master-0-root
uuid: "b2ef36fc26c34975b2271f00d73148f3"
format_stamp: "Formatted at 2025-01-12 20:53:19 on dist-test-slave-3gql"
I20250112 20:53:19.014379 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:19.021152 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:19.021641 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:19.032915 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43453
I20250112 20:53:19.032929 10641 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43453 every 8 connection(s)
I20250112 20:53:19.034068 10642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:19.037856 10642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3: Bootstrap starting.
I20250112 20:53:19.038909 10642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:19.040185 10642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3: No bootstrap required, opened a new log
I20250112 20:53:19.040818 10642 raft_consensus.cc:357] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ef36fc26c34975b2271f00d73148f3" member_type: VOTER }
I20250112 20:53:19.040935 10642 raft_consensus.cc:383] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:19.040966 10642 raft_consensus.cc:738] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2ef36fc26c34975b2271f00d73148f3, State: Initialized, Role: FOLLOWER
I20250112 20:53:19.041127 10642 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ef36fc26c34975b2271f00d73148f3" member_type: VOTER }
I20250112 20:53:19.041246 10642 raft_consensus.cc:397] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:19.041280 10642 raft_consensus.cc:491] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:19.041326 10642 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:19.042824 10642 raft_consensus.cc:513] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ef36fc26c34975b2271f00d73148f3" member_type: VOTER }
I20250112 20:53:19.042977 10642 leader_election.cc:304] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2ef36fc26c34975b2271f00d73148f3; no voters:
I20250112 20:53:19.043323 10642 leader_election.cc:290] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:19.043442 10645 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:19.043867 10645 raft_consensus.cc:695] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 1 LEADER]: Becoming Leader. State: Replica: b2ef36fc26c34975b2271f00d73148f3, State: Running, Role: LEADER
I20250112 20:53:19.044127 10642 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250112 20:53:19.044106 10645 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ef36fc26c34975b2271f00d73148f3" member_type: VOTER }
I20250112 20:53:19.045176 10647 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2ef36fc26c34975b2271f00d73148f3. Latest consensus state: current_term: 1 leader_uuid: "b2ef36fc26c34975b2271f00d73148f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ef36fc26c34975b2271f00d73148f3" member_type: VOTER } }
I20250112 20:53:19.045374 10647 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:19.047379 10646 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2ef36fc26c34975b2271f00d73148f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2ef36fc26c34975b2271f00d73148f3" member_type: VOTER } }
I20250112 20:53:19.047523 10646 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [sys.catalog]: This master's current role is: LEADER
I20250112 20:53:19.048595 10653 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250112 20:53:19.049983 10653 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250112 20:53:19.050323 416 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250112 20:53:19.053557 10653 catalog_manager.cc:1348] Generated new cluster ID: ff761ffcc9f14d29b9434952cc9f6d6b
I20250112 20:53:19.053643 10653 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250112 20:53:19.065328 10653 catalog_manager.cc:1371] Generated new certificate authority record
I20250112 20:53:19.066052 10653 catalog_manager.cc:1505] Loading token signing keys...
I20250112 20:53:19.074810 10653 catalog_manager.cc:5899] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3: Generated new TSK 0
I20250112 20:53:19.075032 10653 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250112 20:53:19.080216 10597 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:53:19.082892 416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250112 20:53:19.085474 10669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:19.085636 10670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250112 20:53:19.086243 10672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250112 20:53:19.086531 416 server_base.cc:1034] running on GCE node
I20250112 20:53:19.086767 416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250112 20:53:19.086817 416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250112 20:53:19.086853 416 hybrid_clock.cc:648] HybridClock initialized: now 1736715199086849 us; error 0 us; skew 500 ppm
I20250112 20:53:19.086995 416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250112 20:53:19.088136 416 webserver.cc:458] Webserver started at http://127.0.104.1:38777/ using document root <none> and password file <none>
I20250112 20:53:19.088339 416 fs_manager.cc:362] Metadata directory not provided
I20250112 20:53:19.088404 416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250112 20:53:19.088531 416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250112 20:53:19.089345 416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/ts-0-root/instance:
uuid: "3498c4455e924d49ad6f4336e2a9709c"
format_stamp: "Formatted at 2025-01-12 20:53:19 on dist-test-slave-3gql"
I20250112 20:53:19.091733 416 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20250112 20:53:19.093223 10677 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:19.093530 416 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250112 20:53:19.093617 416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/ts-0-root
uuid: "3498c4455e924d49ad6f4336e2a9709c"
format_stamp: "Formatted at 2025-01-12 20:53:19 on dist-test-slave-3gql"
I20250112 20:53:19.093721 416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskRSg17H/test-tmp/txn_write_ops-itest.0.TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads.1736715060603202-416-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250112 20:53:19.110013 416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250112 20:53:19.110682 416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250112 20:53:19.112438 416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250112 20:53:19.112524 416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:19.112614 416 ts_tablet_manager.cc:610] Registered 0 tablets
I20250112 20:53:19.112666 416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250112 20:53:19.133596 416 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40541
I20250112 20:53:19.133620 10746 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40541 every 8 connection(s)
I20250112 20:53:19.135004 10747 heartbeater.cc:346] Connected to a master server at 127.0.104.62:43453
I20250112 20:53:19.135098 10747 heartbeater.cc:463] Registering TS with master...
I20250112 20:53:19.135388 10747 heartbeater.cc:510] Master 127.0.104.62:43453 requested a full tablet report, sending...
I20250112 20:53:19.136045 10597 ts_manager.cc:194] Registered new tserver with Master: 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1:40541)
I20250112 20:53:19.136720 416 internal_mini_cluster.cc:371] 1 TS(s) registered with all masters after 0.002275478s
I20250112 20:53:19.137020 10597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48420
I20250112 20:53:19.137655 416 test_util.cc:274] Using random seed: -2071640206
I20250112 20:53:19.147493 10597 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48432:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250112 20:53:19.160272 10709 tablet_service.cc:1467] Processing CreateTablet for tablet b946f693901f48a29aa4fc8cd17526c4 (DEFAULT_TABLE table=test-workload [id=d9d2898a8bc34a4eb780ecbf7a03bf38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:19.160390 10705 tablet_service.cc:1467] Processing CreateTablet for tablet 7c0aa8a465824fcaa9c22ab9bea100a7 (DEFAULT_TABLE table=test-workload [id=d9d2898a8bc34a4eb780ecbf7a03bf38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:19.160352 10707 tablet_service.cc:1467] Processing CreateTablet for tablet f6313f82dff34b1b84a79cda98b6eb12 (DEFAULT_TABLE table=test-workload [id=d9d2898a8bc34a4eb780ecbf7a03bf38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:19.160770 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b946f693901f48a29aa4fc8cd17526c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:19.160980 10707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6313f82dff34b1b84a79cda98b6eb12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:19.161175 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c0aa8a465824fcaa9c22ab9bea100a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:19.165239 10757 tablet_bootstrap.cc:492] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c: Bootstrap starting.
I20250112 20:53:19.166500 10757 tablet_bootstrap.cc:654] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:19.167785 10757 tablet_bootstrap.cc:492] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c: No bootstrap required, opened a new log
I20250112 20:53:19.167881 10757 ts_tablet_manager.cc:1397] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:19.168355 10757 raft_consensus.cc:357] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.168471 10757 raft_consensus.cc:383] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:19.168501 10757 raft_consensus.cc:738] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Initialized, Role: FOLLOWER
I20250112 20:53:19.168676 10757 consensus_queue.cc:260] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.168805 10757 raft_consensus.cc:397] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:19.168844 10757 raft_consensus.cc:491] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:19.168892 10757 raft_consensus.cc:3054] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:19.170493 10757 raft_consensus.cc:513] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.170682 10757 leader_election.cc:304] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3498c4455e924d49ad6f4336e2a9709c; no voters:
I20250112 20:53:19.171012 10757 leader_election.cc:290] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:19.171131 10759 raft_consensus.cc:2798] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:19.171603 10747 heartbeater.cc:502] Master 127.0.104.62:43453 was elected leader, sending a full tablet report...
I20250112 20:53:19.171635 10757 ts_tablet_manager.cc:1428] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:19.171864 10757 tablet_bootstrap.cc:492] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c: Bootstrap starting.
I20250112 20:53:19.172137 10759 raft_consensus.cc:695] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Becoming Leader. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Running, Role: LEADER
I20250112 20:53:19.172376 10759 consensus_queue.cc:237] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.173190 10757 tablet_bootstrap.cc:654] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:19.174873 10757 tablet_bootstrap.cc:492] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c: No bootstrap required, opened a new log
I20250112 20:53:19.174839 10597 catalog_manager.cc:5526] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:19.174996 10757 ts_tablet_manager.cc:1397] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:19.175655 10757 raft_consensus.cc:357] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.175801 10757 raft_consensus.cc:383] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:19.175859 10757 raft_consensus.cc:738] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Initialized, Role: FOLLOWER
I20250112 20:53:19.176026 10757 consensus_queue.cc:260] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.176167 10757 raft_consensus.cc:397] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:19.176226 10757 raft_consensus.cc:491] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:19.176309 10757 raft_consensus.cc:3054] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:19.178308 10757 raft_consensus.cc:513] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.178498 10757 leader_election.cc:304] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3498c4455e924d49ad6f4336e2a9709c; no voters:
I20250112 20:53:19.178666 10757 leader_election.cc:290] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:19.178723 10759 raft_consensus.cc:2798] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:19.178853 10759 raft_consensus.cc:695] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Becoming Leader. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Running, Role: LEADER
I20250112 20:53:19.179059 10757 ts_tablet_manager.cc:1428] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:19.179085 10759 consensus_queue.cc:237] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.179275 10757 tablet_bootstrap.cc:492] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c: Bootstrap starting.
I20250112 20:53:19.180850 10757 tablet_bootstrap.cc:654] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:19.181267 10597 catalog_manager.cc:5526] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:19.182521 10757 tablet_bootstrap.cc:492] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c: No bootstrap required, opened a new log
I20250112 20:53:19.182638 10757 ts_tablet_manager.cc:1397] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:53:19.183216 10757 raft_consensus.cc:357] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.183363 10757 raft_consensus.cc:383] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:19.183431 10757 raft_consensus.cc:738] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Initialized, Role: FOLLOWER
I20250112 20:53:19.183619 10757 consensus_queue.cc:260] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.183815 10757 raft_consensus.cc:397] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:19.183904 10757 raft_consensus.cc:491] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:19.183995 10757 raft_consensus.cc:3054] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:19.185636 10757 raft_consensus.cc:513] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.185827 10757 leader_election.cc:304] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3498c4455e924d49ad6f4336e2a9709c; no voters:
I20250112 20:53:19.185971 10757 leader_election.cc:290] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:19.186058 10761 raft_consensus.cc:2798] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:19.186209 10761 raft_consensus.cc:695] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Becoming Leader. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Running, Role: LEADER
I20250112 20:53:19.186409 10757 ts_tablet_manager.cc:1428] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:19.186434 10761 consensus_queue.cc:237] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:19.188249 10596 catalog_manager.cc:5526] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } health_report { overall_health: HEALTHY } } }
W20250112 20:53:19.197558 10750 txn_manager_proxy_rpc.cc:150] re-attempting BeginTransaction request to one of TxnManagers
I20250112 20:53:20.094197 10596 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48446:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
I20250112 20:53:20.100941 10705 tablet_service.cc:1467] Processing CreateTablet for tablet 9f2c35b606f04da9a8ff6d962d270627 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6eafe76b2bb74c2193b0cbaffee59d08]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20250112 20:53:20.101387 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f2c35b606f04da9a8ff6d962d270627. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:20.105947 10777 tablet_bootstrap.cc:492] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: Bootstrap starting.
I20250112 20:53:20.107473 10777 tablet_bootstrap.cc:654] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:20.108850 10777 tablet_bootstrap.cc:492] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: No bootstrap required, opened a new log
I20250112 20:53:20.108986 10777 ts_tablet_manager.cc:1397] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250112 20:53:20.109553 10777 raft_consensus.cc:357] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.109773 10777 raft_consensus.cc:383] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:20.109869 10777 raft_consensus.cc:738] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Initialized, Role: FOLLOWER
I20250112 20:53:20.110057 10777 consensus_queue.cc:260] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.110179 10777 raft_consensus.cc:397] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:20.110256 10777 raft_consensus.cc:491] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:20.110352 10777 raft_consensus.cc:3054] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:20.111903 10777 raft_consensus.cc:513] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.112092 10777 leader_election.cc:304] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3498c4455e924d49ad6f4336e2a9709c; no voters:
I20250112 20:53:20.112425 10777 leader_election.cc:290] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:20.112592 10779 raft_consensus.cc:2798] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:20.113030 10779 raft_consensus.cc:695] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Becoming Leader. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Running, Role: LEADER
I20250112 20:53:20.113090 10777 ts_tablet_manager.cc:1428] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:20.113329 10779 consensus_queue.cc:237] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.114558 10780 tablet_replica.cc:440] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } } }
I20250112 20:53:20.114750 10780 tablet_replica.cc:443] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:53:20.114929 10781 tablet_replica.cc:440] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: TxnStatusTablet state changed. Reason: New leader 3498c4455e924d49ad6f4336e2a9709c. Latest consensus state: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } } }
I20250112 20:53:20.115165 10781 tablet_replica.cc:443] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: This TxnStatusTablet replica's current role is: LEADER
I20250112 20:53:20.115651 10596 catalog_manager.cc:5526] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:20.116755 10783 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20250112 20:53:20.116878 10783 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20250112 20:53:20.755368 416 test_util.cc:274] Using random seed: -2070022504
I20250112 20:53:20.766327 10591 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48456:
name: "default.second_table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20250112 20:53:20.774546 10707 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb164520a774e1f8dddbf0217a9e542 (DEFAULT_TABLE table=default.second_table [id=ed3cb0efacd94700b48591b8268ff8a0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:20.774560 10709 tablet_service.cc:1467] Processing CreateTablet for tablet 453eab9d08fe4c8cb76129f089cfd0ca (DEFAULT_TABLE table=default.second_table [id=ed3cb0efacd94700b48591b8268ff8a0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:20.774749 10705 tablet_service.cc:1467] Processing CreateTablet for tablet 60d61117ab3049378289c360873dd806 (DEFAULT_TABLE table=default.second_table [id=ed3cb0efacd94700b48591b8268ff8a0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20250112 20:53:20.775027 10707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eb164520a774e1f8dddbf0217a9e542. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:20.775197 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 453eab9d08fe4c8cb76129f089cfd0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:20.775362 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d61117ab3049378289c360873dd806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250112 20:53:20.779626 10818 tablet_bootstrap.cc:492] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c: Bootstrap starting.
I20250112 20:53:20.780963 10818 tablet_bootstrap.cc:654] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:20.782316 10818 tablet_bootstrap.cc:492] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c: No bootstrap required, opened a new log
I20250112 20:53:20.782428 10818 ts_tablet_manager.cc:1397] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:20.782982 10818 raft_consensus.cc:357] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.783143 10818 raft_consensus.cc:383] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:20.783180 10818 raft_consensus.cc:738] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Initialized, Role: FOLLOWER
I20250112 20:53:20.783351 10818 consensus_queue.cc:260] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.783468 10818 raft_consensus.cc:397] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:20.783527 10818 raft_consensus.cc:491] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:20.783607 10818 raft_consensus.cc:3054] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:20.785355 10818 raft_consensus.cc:513] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.785518 10818 leader_election.cc:304] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3498c4455e924d49ad6f4336e2a9709c; no voters:
I20250112 20:53:20.785643 10818 leader_election.cc:290] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:20.785717 10807 raft_consensus.cc:2798] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:20.785826 10807 raft_consensus.cc:695] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Becoming Leader. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Running, Role: LEADER
I20250112 20:53:20.786044 10818 ts_tablet_manager.cc:1428] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:20.786049 10807 consensus_queue.cc:237] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.786268 10818 tablet_bootstrap.cc:492] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c: Bootstrap starting.
I20250112 20:53:20.787786 10818 tablet_bootstrap.cc:654] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:20.788172 10594 catalog_manager.cc:5526] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:20.789193 10818 tablet_bootstrap.cc:492] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c: No bootstrap required, opened a new log
I20250112 20:53:20.789319 10818 ts_tablet_manager.cc:1397] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:20.789898 10818 raft_consensus.cc:357] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.790014 10818 raft_consensus.cc:383] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:20.790081 10818 raft_consensus.cc:738] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Initialized, Role: FOLLOWER
I20250112 20:53:20.790287 10818 consensus_queue.cc:260] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.790413 10818 raft_consensus.cc:397] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:20.790478 10818 raft_consensus.cc:491] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:20.790549 10818 raft_consensus.cc:3054] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:20.793032 10818 raft_consensus.cc:513] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.793227 10818 leader_election.cc:304] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3498c4455e924d49ad6f4336e2a9709c; no voters:
I20250112 20:53:20.793355 10818 leader_election.cc:290] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:20.793421 10807 raft_consensus.cc:2798] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:20.793530 10807 raft_consensus.cc:695] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Becoming Leader. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Running, Role: LEADER
I20250112 20:53:20.793747 10818 ts_tablet_manager.cc:1428] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:20.793720 10807 consensus_queue.cc:237] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.794047 10818 tablet_bootstrap.cc:492] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c: Bootstrap starting.
I20250112 20:53:20.795490 10818 tablet_bootstrap.cc:654] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c: Neither blocks nor log segments found. Creating new log.
I20250112 20:53:20.795742 10594 catalog_manager.cc:5526] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:20.797094 10818 tablet_bootstrap.cc:492] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c: No bootstrap required, opened a new log
I20250112 20:53:20.797206 10818 ts_tablet_manager.cc:1397] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250112 20:53:20.797705 10818 raft_consensus.cc:357] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.797837 10818 raft_consensus.cc:383] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250112 20:53:20.797895 10818 raft_consensus.cc:738] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Initialized, Role: FOLLOWER
I20250112 20:53:20.798081 10818 consensus_queue.cc:260] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.798228 10818 raft_consensus.cc:397] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250112 20:53:20.798290 10818 raft_consensus.cc:491] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250112 20:53:20.798372 10818 raft_consensus.cc:3054] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 0 FOLLOWER]: Advancing to term 1
I20250112 20:53:20.800566 10818 raft_consensus.cc:513] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.800768 10818 leader_election.cc:304] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3498c4455e924d49ad6f4336e2a9709c; no voters:
I20250112 20:53:20.800920 10818 leader_election.cc:290] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250112 20:53:20.800983 10807 raft_consensus.cc:2798] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Leader election won for term 1
I20250112 20:53:20.801126 10807 raft_consensus.cc:695] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Becoming Leader. State: Replica: 3498c4455e924d49ad6f4336e2a9709c, State: Running, Role: LEADER
I20250112 20:53:20.801352 10818 ts_tablet_manager.cc:1428] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250112 20:53:20.801323 10807 consensus_queue.cc:237] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } }
I20250112 20:53:20.803218 10594 catalog_manager.cc:5526] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3498c4455e924d49ad6f4336e2a9709c (127.0.104.1). New cstate: current_term: 1 leader_uuid: "3498c4455e924d49ad6f4336e2a9709c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3498c4455e924d49ad6f4336e2a9709c" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40541 } health_report { overall_health: HEALTHY } } }
I20250112 20:53:21.017343 416 tablet_server.cc:178] TabletServer@127.0.104.1:0 shutting down...
I20250112 20:53:21.025403 416 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250112 20:53:21.026315 416 tablet_replica.cc:331] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c: stopping tablet replica
I20250112 20:53:21.026556 416 raft_consensus.cc:2238] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.026732 416 raft_consensus.cc:2267] T 60d61117ab3049378289c360873dd806 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.027550 416 tablet_replica.cc:331] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c: stopping tablet replica
I20250112 20:53:21.027683 416 raft_consensus.cc:2238] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.027808 416 raft_consensus.cc:2267] T 7c0aa8a465824fcaa9c22ab9bea100a7 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.028366 416 tablet_replica.cc:331] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c: stopping tablet replica
I20250112 20:53:21.028491 416 raft_consensus.cc:2238] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.028632 416 raft_consensus.cc:2267] T b946f693901f48a29aa4fc8cd17526c4 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.029276 416 tablet_replica.cc:331] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c: stopping tablet replica
I20250112 20:53:21.029421 416 raft_consensus.cc:2238] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.029554 416 raft_consensus.cc:2267] T f6313f82dff34b1b84a79cda98b6eb12 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.030131 416 tablet_replica.cc:331] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c: stopping tablet replica
I20250112 20:53:21.030257 416 raft_consensus.cc:2238] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.030391 416 raft_consensus.cc:2267] T 9f2c35b606f04da9a8ff6d962d270627 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.030910 416 tablet_replica.cc:331] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c: stopping tablet replica
I20250112 20:53:21.031028 416 raft_consensus.cc:2238] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.031153 416 raft_consensus.cc:2267] T 5eb164520a774e1f8dddbf0217a9e542 P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.031848 416 tablet_replica.cc:331] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c: stopping tablet replica
I20250112 20:53:21.031968 416 raft_consensus.cc:2238] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.032086 416 raft_consensus.cc:2267] T 453eab9d08fe4c8cb76129f089cfd0ca P 3498c4455e924d49ad6f4336e2a9709c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.048709 416 tablet_server.cc:195] TabletServer@127.0.104.1:0 shutdown complete.
I20250112 20:53:21.061261 416 master.cc:537] Master@127.0.104.62:43453 shutting down...
I20250112 20:53:21.069243 416 raft_consensus.cc:2238] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 1 LEADER]: Raft consensus shutting down.
I20250112 20:53:21.069475 416 raft_consensus.cc:2267] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250112 20:53:21.069586 416 tablet_replica.cc:331] T 00000000000000000000000000000000 P b2ef36fc26c34975b2271f00d73148f3: stopping tablet replica
I20250112 20:53:21.073063 416 master.cc:559] Master@127.0.104.62:43453 shutdown complete.
[ OK ] TxnOpDispatcherITest.TestSeparateBeginCommitTestWorkloads (2080 ms)
[----------] 18 tests from TxnOpDispatcherITest (81055 ms total)
[----------] Global test environment tear-down
[==========] 26 tests from 2 test suites ran. (140404 ms total)
[ PASSED ] 25 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] TxnOpDispatcherITest.TxnWriteWhileReplicaDeleted
1 FAILED TEST
YOU HAVE 1 DISABLED TEST
I20250112 20:53:21.084993 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2367: suppressed but not reported on 9 messages since previous log ~23 seconds ago
I20250112 20:53:21.085058 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 1 messages since previous log ~12 seconds ago
I20250112 20:53:21.085091 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/transactions/txn_status_manager.cc:765: suppressed but not reported on 99 messages since previous log ~12 seconds ago
I20250112 20:53:21.085121 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet_replica.cc:1307: suppressed but not reported on 2 messages since previous log ~3 seconds ago
I20250112 20:53:21.085180 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/master.cc:440: suppressed but not reported on 3 messages since previous log ~7 seconds ago
I20250112 20:53:21.085269 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/txn_manager_proxy_rpc.cc:150: suppressed but not reported on 6 messages since previous log ~1 seconds ago